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1.  SUMMARY 


Formal  Verification  (FV)  ensures  that  mission-essential  software  is  free  from  disruptive 
errors  and  security  vulnerabilities,  but  requires  human  experts  that  can  be  quickly  overwhelmed 
by  the  increasing  number,  size,  and  complexity  of  software  systems.  The  Defense  Advanced 
Research  Project  Agency  (DARPA)  Crowd  Sourced  Fonnal  Verification  (CSFV)  program  has 
been  building  games  that  recast  FV  problems  into  puzzles  to  make  these  problems  more 
accessible,  increasing  the  manpower  to  construct  FV  proofs.  Much  progress  has  been  made  in  the 
CSFV  program  to  date;  however,  it  has  become  clear  that  it’s  a  struggle  to  make  engaging  games 
that  foster  human  participation  and  effectively  address  formal  verification  problems.  To  augment 
this  work — and  complete  the  evaluation  of  a  game-based  approach — DARPA  needed  a  team  to 
maintain  and  extend  the  CSFV  website,  perform  additional  game  analytics,  and  investigate 
alternative  incentive  methods  to  assess  other  approaches  to  crowd  sourced  software  verification. 
To  address  this  need,  we  conducted  a  CSFV  augmentation  effort  (CSFV-A),  in  which  we: 

•  Managed  the  Verigames  website  and  integration  framework  through  the  close  of  the  program, 
updating  that  website  to  incorporate  Phase  2  games  and  enabling  the  evaluation  of  this  new 
game  set  as  a  tool  to  reduce  the  cost  of  fonnal  verification 

•  Managed  collaboration  activities  across  the  CSFV  program  (including  cross-team  integration 
meetings) 

•  Expanded  our  analytics  efforts  for  CSFV;  in  addition  to  expert  game  design  analysis  and  user 
studies,  we  analyzed  the  play  styles  and  motivations  of  game  experts  (gurus)  using  cognitive 
systems  engineering  (CSE)  techniques 

•  Assessed  pay-for-play  and  other  incentive  mechanisms  for  CSFV,  evaluating  their 
effectiveness  and  cost  in  comparison  to  standard  fonnal  verification  approaches 

•  Marketed  the  CSFV  games  to  engage  participants  in  the  crowd  sourcing  efforts 

•  Managed  the  overall  program  to  provide  high  quality,  on-time  deliverables 


Our  integration  effort  under  CSFV  developed  key  infrastructure  software  to  address 
Technical  Area  2  (TA2)  game  integration  needs,  provided  technical  insights  into  game 
improvements,  investigated  the  impact  of  payment  approaches  compared  to  altruism  and  game- 
based  incentives,  provided  feedback  using  expert  game  design  and  user  analysis  to  improve  the 
playability  and  engagement  value  of  the  games,  provided  an  approach  to  generate  automation 
and  training  to  improve  CSFV  perfonnance  based  on  expert  game  player  analysis,  and 
discovered  that  paid  crowd  sourcing  methods  perform  more  effectively  than  the  current  game- 
based  approach. 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 

1 


2.  INTRODUCTION 


2.1  Problem  Description 

Formal  Verification  (FV)  ensures  that  mission-essential  software  is  free  from  disruptive 
errors  and  security  vulnerabilities,  but  requires  human  experts  that  can  be  quickly  overwhelmed 
by  the  increasing  number,  size,  and  complexity  of  software  systems.  DARPA’s  Crowd  Sourced 
Formal  Verification  (CSFV)  program  has  been  building  games  that  recast  FV  problems  into 
puzzles  to  make  these  problems  more  accessible,  increasing  the  manpower  to  construct  FV 
proofs.  To  date,  in  the  CSFV  program,  much  progress  has  been  made — however,  it  has  become 
clear  that  it’s  a  struggle  to  simultaneously  make  engaging  games  that  foster  human  participation, 
and  can  effectively  address  fonnal  verification  problems.  To  augment  this  work — and  complete 
the  evaluation  of  a  game-based  approach — DARPA  needed  a  team  to  maintain  and  extend  the 
CSFV  website,  perform  additional  game  analytics,  and  investigate  alternative  incentive  methods 
to  assess  other  approaches  to  crowd  sourced  software  verification. 

2.2  Technical  Approach 

We  conducted  a  CSFV  augmentation  effort  (CSFV-A)  that  addressed  these  needs.  We 
successfully  managed  the  Verigames  website  and  integration  framework  through  the  close  of  the 
program.  Verigames.com  was  a  single  entry  point  to  access  all  CSFV  games  developed  under  the 
DARPA  program.  Through  the  CSFV  program  we  updated  that  website  to  incorporate  Phase  2 
games,  enabling  the  evaluation  of  a  new  game  set  that  is  designed  for  better  engagement.  We 
worked  with  Appirio’s  TopCoder  service  to  address  extension  and  maintenance  requirements, 
using  the  same  contest-based  approach  to  development  used  in  previous  CSFV  work.  Through 
these  efforts  we  successfully  exploited  the  crowd  resources  that  were  already  familiar  with 
Verigames. 

To  evaluate  these  games,  we  expanded  our  analytics  efforts  for  CSFV.  We  continued  our 
focus  on  expert  game  designer  analysis  and  user  studies  to  review  the  games  and  provide 
feedback  for  improvements,  and  used  computational  analytics  to  assess  game  play  characteristics 
and  marketing  perfonnance.  We  performed  an  innovative  analysis  of  expert  player  approaches  to 
the  game  and  expert  player  motivations,  using  cognitive  systems  engineering  (CSE)  techniques 
to  observe  and  document  the  game  play  of  the  most  significant  contributors  (gurus)  that  were 
available  from  each  game  (unfortunately,  many  of  the  gurus  were  not  available  for  these  events). 

We  explored  and  assessed  alternative  incentive  mechanisms  for  CSFV.  In  previous  CSFV 
work  by  Technical  Area  1  (TA1)  teams,  the  focus  was  on  a  combination  of  altruism  and 
engagement  value,  designing  game-based  mechanisms  that  attract  citizen  scientists  wishing  to 
contribute  to  fonnal  verification  proofs.  In  Phase  2,  we  enabled  a  pay-to-play  mechanism  to 
address  CSFV  needs,  testing  the  ability  for  limited  payments  in  traditional  crowd  sourcing 
environments  to  encourage  greater  participation  and  contributions  to  CSFV  (via  Amazon’s 
Mechanical  Turk).  We  also  explored  a  contest-oriented  incentive  mechanism  in  which  players 
who  perfonn  particularly  strongly  in  the  games  (produce  the  highest  scores)  gained  access  to 
extra  community  events  and  acknowledgement  on  the  website.  Finally,  we  investigated  the  use 
of  TopCoder  Bug  Hunts  as  a  means  to  drive  more  game  play  traffic  on  the  website  as  another 
pay-to-play  incentive  approach.  Each  of  the  pay-to-play  mechanisms  explored  had  significant 
impacts  on  play  characteristics  and  timeline,  and  were  ultimately  successful  tools  to  enhance 
crowdsourcing  contributions.  The  contest-oriented  incentives  did  not  have  a  significant  impact 
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on  participation;  however,  that  may  be  because  we  were  not  able  to  provide  payment-based 
incentives  in  that  effort. 

We  made  a  few  changes  to  this  technical  approach  during  the  effort.  We  changed  the  contest 
from  a  monetary-based  reward  to  a  non-monetary,  community-based  reward  to  make  the  contest 
feasible  to  execute,  while  maintaining  the  potential  for  increased  play  statistics.  We  also 
identified  the  need  to  maintain  a  basic  Verigames  website  beyond  the  end  of  the  program,  so  we 
designed  and  generated  a  basic  website  that  does  not  host  the  games,  but  rather  points  to  games 
hosted  by  the  TA1  teams. 

2.3  Technical  Objectives 

Under  this  effort,  Charles  River  Analytics  executed  a  Crowd  Sourced  Fonnal  Verification 
(CSFV)  augmentation  effort.  Our  first  objective  was  to  manage  the  Verigames  website  and 
integration  framework  through  the  close  of  the  program,  updating  the  site  to  incorporate  Phase  2 
games,  and  enabling  the  evaluation  of  this  new  game  set  as  a  tool  to  reduce  the  cost  of  formal 
verification.  To  evaluate  these  games,  our  next  objective  was  to  continue  and  expand  our 
analytics  efforts  for  CSFV.  In  addition  to  expert  game  design  analysis,  user  studies,  and 
computational  analytics,  we  analyzed  the  play  styles  of  game  experts  {guru ’s)  using  cognitive 
systems  engineering  (CSE)  techniques  to  observe  and  document  their  approaches.  Next,  we 
assessed  pay-for-play  incentive  mechanisms  for  CSFV,  evaluating  their  effectiveness  and  cost  in 
comparison  to  standard  fonnal  verification  approaches.  Specifically,  we  explored  using 
Mechanical  Turk  to  motivate  accomplishment,  and  contests  to  motivate  participation  in  the 
game-based  approach.  We  also  maintained  targeted  marketing  efforts  to  foster  crowd 
participation  in  CSFV. 

The  purpose  of  the  DARPA  CSFV  program  is  to  investigate  low-cost  alternatives  to  software 
verification  that  can  reduce  the  reliance  on  expensive  expert  computer  scientists  and 
mathematicians.  The  CSFV-Augmentation  (CSFV-A)  effort  (Phase  2)  supported  this  objective  in 
the  following  ways: 

•  Provide  a  robust  and  attractive  web-based  delivery  platform  for  game-based  formal  verification  tools. 
We  updated  the  Verigames  website  to  incorporate  the  Phase  2  games  being  developed  by  TA1 
performers,  and  provided  those  games  in  a  robust  and  attractive  format  to  engender  crowd  participation. 

•  Explore  game  analytic  methods  to  enhance  crowd  participation  in  gameplay.  We  used  expert  analysis, 
computational  analysis,  and  user  evaluation  to  make  recommendations  to  TA1  performers  to  improve 
the  playability  of  the  games.  Phase  2  also  included  an  evaluation  of  experts’  (gurus)  play  styles  to 
recommend  improvements  to  tutorials  to  generate  more  experts,  and  potential  approaches  to  develop 
automated  agents  that  mimic  those  play  styles. 

•  Explore  contests  to  enhance  crowd  participation  in  game  play.  We  explored  the  use  of  contests  to  drive 
increased  participation  in  game  play.  Specifically,  a  single  contest  was  held,  with  manageable  prizes, 
and  data  was  collected  on  the  impact  this  contest  has  on  player  participation. 

•  Assess  paid  crowd  sourcing  as  a  low-cost  alternative  to  expert  formal  verification.  We  developed  an 
integration  platfomi  for  TA1  performers  with  Amazon’s  Mechanical  Turk  (AMT).  Charles  River 
Analytics  worked  with  TA1  performers  to  integrate  non-game  tasks  into  AMT,  and  tested  the  ability 
for  minimal  payments  to  produce  useful  inputs  to  fonnal  verification  proofs. 

•  Market  key  CSFV  efforts.  During  Phase  2,  Charles  River  Analytics  worked  with  GameDocs  to  market 
Phase  2  game  releases,  Phase  2  game  contests,  and  paid  crowd  sourcing  releases  to  drive  increased 
crowd  participation  in  the  CSFV  effort. 
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3.  METHODS,  ASSUMPTIONS  AND  PROCEDURES 


3.1  Presentations 

We  presented  a  paper  with  Dr.  Drew  Dean  at  the  2015  USENIX  Summit  on  Gaming,  Games, 
and  Gamification  in  Security  Education  (3GSE  ’15)  on  August  1 1,  2015. 


3.2  System  Architecture 

The  primary  focus  of  our  effort  was  to  update,  maintain,  and  host  the  Verigames  website  and 
integration  framework  to  enable  continued  testing  of  Phase  2  games.  The  Verigames  website 
architecture  features  separate  content,  personalization,  persistence,  and  game  support  tiers  that 
are  loosely  connected  by  a  RESTful  API.  This  architecture  allows  for  the  development, 
maintenance,  and  hosting  of  several  games  developed  in  different  technologies  and  for  different 
purposes.  Each  Verigames  Site  game  may  operate  singly,  and  to  integrate  with  the  whole,  can 
asynchronously  call  the  API  to  authenticate  users,  record  scores,  record  game  statistics,  or  pull 
user  profiles.  Figure  1  illustrates  the  architecture  of  the  Verigames  website. 
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Figure  1  Verigames  Website  Logical  Architecture 


The  Verigames.com  infrastructure  operates  on  the  Amazon  Web  Service  (AWS)  platfonn. 
AWS  is  a  class  of  managed  services  offered  by  Amazon.  By  using  the  AWS  cloud,  the 
Verigames  system  benefits  from  the  99.999%  uptime  that  Amazon  provides.  The  Verigames 
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system  also  benefits  from  virtual  machines  (VMs)  offered  at  significantly  lower  cost  than 
dedicated  hardware  and  from  managing  resource  allocation  and  CPU  configuration  and  count  via 
web  services  and  management  dashboards.  This  approach  offered  maximum  flexibility  and 
reliability  at  minimum  cost.  Figure  2  illustrates  the  Verigames.com  AWS  architecture. 


The  program  established  three  environments  using  AWS  cloud  technology: 

•  Production  environment  for  hosting  the  public  site  (Verigames.com) 

•  Testing  environment  (Verigames.org)  for  staging  the  site  for  testing  before  updates  are  posted 
to  production 

•  TA1  development  environment  that  allocates  VMs  on  AWS  for  specific  game  developers 
In  the  production  environment,  the  system  is  configured  as  follows: 

•  Six  large  VMs,  one  for  each  website 

•  Nginx  hosted  on  a  dedicated  small  VM 

•  logaholic  hosted  on  a  dedicated  small  VM 

•  mysql  hosted  on  a  large  VM 

•  Five  medium  VMs,  one  for  each  team’s  backend  and  verification 

•  Amazon  S3  for  storing  photos  and  uploaded  files/attachments  in  the  websites  and  Amazon  SES 
for  sending  emails 

In  the  stage/testing  environment,  the  system  is  configured  as  follows: 
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•  Six  large  VMs,  one  for  each  website 

•  Nginx  hosted  on  a  dedicated  small  VM 

•  Medium  VM  for  Mechanical  Turk  dashboard 

•  Two  extra-large  VMs  to  create  install  DVDs 

•  Ten  small  VMs,  two  for  each  team’s  backend  and  verification 

Ten  micro  and  small  VMs  were  allocated  to  individual  TA1  and  game  developers. 

3.3  Update,  Maintain,  and  Host  Website  and  Framework 

We  maintained  the  Verigames  websites  for  the  CSFV  augmentation  program  to  support 
continued  analysis  of  game-based  crowd  sourcing.  We  analyzed  and  optimized  the  efficiency  of 
the  website,  strategically  addressing  issues  to  ensure  that  download  times  did  not  prevent  a 
successful  CSFV  experience  for  users.  Charles  River  Analytics  performed  this  analysis  and 
improvement  during  the  deployment  cycle,  augmented  by  TopCoder  QA  contests.  We  also 
worked  with  WhiteHat  Securities  to  assess  security  issues  on  the  new  Phase  2  game  pages, 
addressing  security  issues  as  they  were  identified. 

3.3.1  Website  Improvements 

The  Verigames  website  was  updated  to  address  issues  identified  in  integrating  TA1  Phase  2 
games,  enhance  usability,  and  to  incorporate  additional  pages  to: 

•  Market  to  citizen  scientists  to  foster  more  effective  community  support  for  the  scientific 
objectives  of  CSFV 

•  Track  high  scores  and  awards  to  support  a  performance-based  contest 

•  Better  manage  the  website 

We  also  integrated  the  Mechanical  Turk  platform  and  implemented  tags  for  contest  winners 
after  the  high  scorer  contest. 

The  Citizen  Science  page  portrays  materials  in  a  manner  that  attracts  citizen  scientists,  with  a 
focus  on  the  science  and  advancement  generated  by  the  community  (as  opposed  to  a  game¬ 
centric  approach).  We  believed  this  approach  would  be  more  attractive  for  common  community 
members  participating  in  CSFV.  Our  initial  citizen  science  page  was  made  available  as  a  secured 
page  (via  link)  on  May  6,  2015. 

Specifically,  we  ensured  all  the  content  on  the  verigames.com  site  was  up  to  date  and 
relevant  to  Phase  2.  We  converted  existing  2014  videos  into  Phase  2  content  (e.g.,  referring  to 
Phase  2  games).  We  also  considered  podcasts,  articles,  and  blog  posts  for  previously  generated 
but  unused  content. 

We  compiled  all  relevant  scientific  information  for  the  Citizen  Science  page  and  provided 
content  and  instructions  for  adding  and  fonnatting  the  content.  We  added  sharing  icons  to  the 
Citizen  Science  page  and  a  Play  Now  call  to  action.  We  integrated  science  links  on  the  play 
pages  that  display  the  Citizen  Science  page.  We  updated  the  Citizen  Science  page  throughout  the 
effort  with  new  media  articles,  blog  posts,  and  videos,  as  well  as  scholarly  articles,  and  we  added 
a  feed  from  the  Monster  Proof  game  with  analyzed  lines  of  code.  We  rewrote  the  About  Us 
section  of  the  website  to  improve  the  PR  story  and  provide  additional  information  for  press 
outlets.  We  added  content  to  the  News  page  and  Forum  to  promote  community  events,  press 
events,  and  new  site  content  and  frequently  updated  the  drop-down  banner  to  promote 
community  events  and  contests.  We  also  changed  the  layout  of  the  home  page  to  better  showcase 
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its  content  and  we  added  new  content  to  show  the  results  of  games  that  provided  live  statistics  on 
formal  verification  results. 

Once  changes  were  implemented,  we  reviewed  the  website  and  entered  Jira  tickets  to  correct 
errors  and  bugs,  and  we  monitored  the  Forum  and  email  for  customer  support  queries. 

3.3.2  Friends  &  Family  Release 

We  began  our  effort  by  updating  and  creating  virtual  machines  on  the  production  server  for 
each  of  the  Phase  2  games,  accessible  via  non-indexed  web  addresses.  We  fixed  issues  as  they 
arose  to  allow  the  games  to  be  hosted  on  directly  linked  sites  for  the  Friends  &  Family  Beta  test. 
See  Appendix  A  for  a  summary  of  technical  accomplishments  in  support  of  this  release. 

We  gathered  lists  of  usernames  from  each  of  the  TA1  teams  to  be  invited  to  participate  in  the 
Friends  &  Family  Beta  test  and  added  the  stored  email  addresses  for  those  names  to  lists  in 
MailChimp,  so  that  user  infonnation  would  remain  blind  to  the  TA1  teams.  We  sent  emails  for 
the  event  that  included  art  and  a  description  of  each  game. 

We  were  prepared  for  the  Friends  &  Family  Beta  release  on  March  31,  2015  for  four  of  the 
five  TA1  teams,  but  were  delayed  due  to  the  need  to  approve  game  materials  for  public  release. 
We  coordinated  the  submission  of  assets,  text,  and  audio  materials  from  the  games  to  DARPA 
for  approval.  Once  they  were  approved,  we  waited  until  April  7,  2015  for  the  fifth  game  to  be 
ready.  The  Friends  &  Family  release  went  live  with  four  of  the  five  games  on  April  8,  2015 
(Dynamakr,  Ghost  Map:  Hyperspace,  Monsterproof,  and  Paradox),  with  the  fifth  game  released 
on  April  10,  2015  (Binary  Fission). 

3.3.3  Website  Security  Test 

We  had  an  early  version  of  the  public  release  in  place  for  April  3 1,  2015  to  run  the  WhiteHat 
test  for  website  security.  Our  WhiteHat  evaluation  identified  minor  issues  unlikely  to  cause 
security  leaks.  We  decided  not  to  address  these  issues,  as  the  risk  was  low  compared  to  the 
expense  for  a  resolution. 

Specifically,  WhiteHat  identified  a  single  (albeit  repeated)  issue  for  Verigames  security — 
specifically,  an  Insufficient  Transport  Layer  Protection  fault.  Several  pages  used  transport  layer 
protection  (HTTPS),  but  loaded  content  that  did  not  use  transport  layer  protection  (HTTP). 
Insufficient  transport  layer  protection  allows  communication  to  be  exposed  to  untrusted  third 
parties,  providing  an  attack  vector  to  compromise  a  web  application  and/or  steal  sensitive 
infonnation.  When  the  transport  layer  is  not  encrypted,  all  communication  between  the  website 
and  the  client  is  sent  in  clear  text,  which  leaves  it  open  to  interception,  injection,  and  redirection, 
also  known  as  a  man-in-the-middle/MITM  attack.  An  attacker  may  passively  intercept  the 
communication,  giving  them  access  to  any  sensitive  data  that  is  being  transmitted,  such  as 
usernames  and  passwords.  An  attacker  may  also  actively  inject/remove  content  from  the 
communication,  allowing  the  attacker  to  forge  and  omit  infonnation,  inject  malicious  scripting, 
or  cause  the  client  to  access  remote,  untrusted  content.  An  attacker  may  also  redirect  the 
communication  so  that  the  website  and  client  are  no  longer  communicating  with  each  other,  but 
instead  are  unknowingly  communicating  with  the  attacker  in  the  context  of  the  other  trusted 
party. 

The  pages  identified  as  vulnerable  contained  resources  categorized  as  passive  mixed  content 
that  were  displayed  to  the  user  and  had  minimal  impact  on  the  behavior  of  the  page.  These 
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include  audio  fdes,  images,  videos,  and  other  objects.  This  issue  was  not  exploited  by  third 
parties  and  caused  no  problems  on  the  website. 

3.3.4  Official  Public  Release 

All  Phase  2  games  were  ready  for  play  by  the  public  on  May  27,  2015.  The  verigames.com 
website  was  ready,  including  logins,  forums,  and  links  to  the  new  games.  The  official  public 
release  supported  contests  and  observations  that  occurred  later  in  the  effort. 

3.3.5  Play  Statistics 

We  tracked  play  statistics  for  registered  and  anonymous  users  throughout  the  period,  as 
shown  in  Figure  3  and  Figure  4. 


m  oo 
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•Unique  Logged-in  Users 


Viewed  by  Logged-in  Users 


Figure  3  Statistics  for  Registered  Verigames  Users 
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Viewed  by  Anonymous  Users 


Figure  4  Statistics  for  Anonymous  Verigames  Users 


A  significant  spike  occurred  at  the  end  of  May,  when  the  Phase  2  games  went  live  with  the 
public  release  and  a  DARPA  news  item.  Smaller  spikes  were  associated  with  our  marketing 
efforts,  such  as  the  spike  around  June  25  due  to  marketing  the  beginning  of  the  guru  search,  but 
the  traffic  was  low  in  comparison  to  our  original  participation  spike.  We  explored  methods  for 
improving  participation  with  our  marketing  consultants,  GameDocs,  as  described  in  Section  3.7. 

In  July,  spikes  occurred  due  to  the  BBC  and  other  press  releases,  and  the  beginning  of  the 
Verigames  contest.  In  August  and  September,  traffic  lowered  as  the  marketing  effort  ended,  with 
some  spikes  due  to  the  TopCoder  Bug  Hunting  events.  The  remainder  of  the  program  had 
numbers  similar  to  those  seen  in  late  August  and  September,  as  there  are  currently  no  planned 
marketing  events  to  increase  numbers.  Numbers  after  September  were  quite  low  in  comparison; 
we  expect  that  his  was  primarily  because  we  ended  the  marketing  effort  in  September. 

3.3.6  Permanent  Website 

We  closed  the  staging  and  productions  servers  at  the  end  of  November  2015  and  created  a 
limited  website  for  pennanent  hosting  that  provides  links  to  the  permanent  BBN,  Washington, 
and  SRI  game  sites.  The  primary  intent  of  this  website  is  to  ensure  that  publications  always  point 
to  something  associated  with  Verigames. 

3.4  Manage  Program  Collaboration 

We  maintained  and  managed  existing  collaboration  efforts  for  CSFV  throughout  the  effort. 
We  maintained  BaseCamp  as  a  collaboration  environment  for  planning  and  engineering 
discussions,  and  Jira  for  reporting  potential  issues  with  the  website  and  integration  framework. 

We  held  weekly  meetings  with  each  TA1  team  to  identify  each  team’s  specific  requirements 
for  the  website  and  integration  framework.  We  held  biweekly  cross-team  meetings  with  Charles 
River,  Appirio,  and  TopCoder  to  discuss  integration  needs,  including  Verigames  and  Mechanical 
Turk  integration.  We  held  game  design  meetings  to  discuss  expert  analysis,  status  of  the  games, 
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and  refinements  to  the  games  to  improve  their  playability.  We  held  weekly  marketing  meetings 
with  GameDocs  to  track  CSFV  marketing  efforts  and  guide  TA1  teams  in  their  marketing 
efforts. 


3.5  Analyze  CSFV  Games 

A  key  facet  of  our  effort  was  continued  analytics  to  help  improve  the  games  and  marketing 
efforts.  To  date,  our  most  successful  analytics  involved  using  game  design  experts  to  review,  in 
detail,  the  game  environment,  gameplay  mechanics,  and  surrounding  narrative  and  content.  We 
continued  to  perfonn  this  task  through  the  close  of  game  development  efforts,  holding  biweekly 
meetings  with  the  TA1  perfonners  to  discuss  game  design  topics  that  can  drive  game 
improvements.  Expert-based  analysis  offers  many  advantages,  producing  straightforward, 
actionable  recommendations  without  the  need  for  large  player  communities,  and  allowing 
analysis  of  factors  that  cannot  be  considered  based  on  pure  data. 

To  better  understand  how  games  might  be  improved,  we  invited  the  top  contributors  (known 
as  gurus)  for  each  of  the  Phase  1  games  to  the  Friends  &  Family  Beta  release  of  the  Phase  2 
games,  providing  an  early  opportunity  for  these  individuals  to  become  guru’s  in  the  new  game 
setting.  We  then  invited  the  best  perfonning  experts  to  in-person  interview  and  observation 
sessions  with  DARPA  and  CSFV  personnel  to  analyze  their  play  style  and  approach.  Top 
performers  were  invited  to  our  facility,  specifically  designed  for  usability  observation,  where  we 
observed  and  recorded  their  game  play  and  asked  targeted  questions  to  understand  how  they 
addressed  the  problems  in  these  games.  A  cognitive  systems  engineering  (CSE)  approach  (Roth 
&  Bisantz,  2013;  Roth  et  al.,  2010)  guided  this  effort  with  best  practices  for  evaluating  expert 
behaviors.  When  performing  this  analysis,  our  primary  objective  was  to  identify  play  style 
elements  that  can  potentially  guide  the  development  of  automated  agents  to  play  the  games  and 
further  improve  the  efficiency  of  fonnal  verification.  A  secondary  objective  was  to  use  these 
play  style  elements  to  guide  the  development  of  training  materials  to  generate  higher  perfonning 
users  across  the  community.  Both  of  these  objectives  can  greatly  enhance  the  contributions  of 
crowd  sourcing  to  formal  verification  processes.  We  held  this  event  in  concert  with  the  PI 
meeting  in  June,  inviting  the  top  contributors  to  that  meeting  for  a  parallel  analysis. 

To  improve  user  attraction  and  retention  on  verigames.com,  we  conducted  a  round  of 
usability  testing  on  the  Phase  2  games.  This  was  a  web-based  usability  testing  session  scheduled 
to  align  with  the  Friends  &  Family  Beta  release,  where  the  focus  was  to  get  rapid 
recommendations  for  improvements  to  implement  for  the  full  release.  YouEye  performed  this 
task. 

We  pursued  computational  analytics  where  possible,  given  the  data  made  available  by  the 
TA1  performers  and  the  marketing  team.  Metrics-based  analytics  focused  on  the  measurable 
aspects  of  the  play  experience  at  different  scales — from  individual  levels  to  play  sessions  to 
careers — and  largely  sought  to  find  common  aspects  of  “successful”  and  “unsuccessful”  play 
patterns  across  the  spectrum  of  factors.  These  factors  included  marketing  efforts,  as  well  as 
tutorial  experience,  level  design  and  features,  sequence  of  level  distribution,  strings  of  successes 
or  failures,  and  meta-game  factors. 
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3.5.1  Expert  Analysis 

We  conducted  an  expert  analysis  of  the  five  Phase  2  games  (Dynamakr,  Binary  Fission, 
Monsterproof,  Paradox,  and  GhostMap:  Hyperspace).  The  full  reports  are  included  in 
Appendices  B  through  F. 

Based  on  this  analysis,  we  determined  that  two  of  the  games  needed  to  refine  their 
terminology  and  the  order  in  which  they  presented  concepts  to  the  player  to  make  sure  that 
players  understand  what  they  are  supposed  to  do  and  what  to  expect  from  their  actions.  All  the 
games  needed  to  refine  their  reward  loops  to  ensure  players  gain  some  personal  value  from  play 
past  the  tutorial.  These  and  other  recommendations  to  improve  the  games  were  addressed  by 
many  of  the  TA1  teams  before  the  official  public  release. 

3.5.2  Usability  Study 

We  worked  with  YouEye  to  execute  a  usability  study  on  the  five  Phase  2  games.  More 
detailed  reports  are  provided  in  Appendices  I  through  M.  Full-scope  video  results  are  available 
online.  The  level  of  detail  for  each  analysis  was  driven  by  the  level  of  interaction  from  the  TA1 
teams  (that  is,  TA1  teams  that  used  the  service  and  requested  analyses  received  more  detailed 
feedback). 

Table  1  provides  a  summary  of  the  key  findings  of  the  usability  study. 


Table  1  YouEye  Key  Findings  for  Phase  2  Games 


Game 

Key  Findings 

Dynamakr 

1.  Initial  intro  and  design  allures  people  to  potential  of  engaging  gameplay.  Maintain  sound  effects  and  galactic 
scale. 

2.  High  success  with  building  patterns,  some  players  missed  the  energy  building  requirement. 

3.  What  makes  a  pattern  better  than  others?  Why  would  you  load  up  on  the  right,  but  not  the  left? 

4.  Unless  paying  close  attention  it  is  very  easy  to  miss  how  pattern  building  relates  to  what  the  Dynamo 
experience  will  be.  This  is  pivotal  to  the  game.  Should  players  be  utilizing  strategy  when  building  patterns  as 
a  way  to  impact  the  Dynamo  shootout?  If  so,  communicate  this  earlier. 

5.  Make  certain  that  Go  Dynamo!  and  Continue  buttons  are  large  and  noticeable,  some  players  missed  them 
initially  which  caused  delays.  Potentially,  remove  the  need  to  click  to  the  next  space,  as  long  as  there  is 
some  transition  demo  to  introduce  the  Dynamo  and  its  levels 

6.  The  majority  of  players  suggested  leveraging  more  tooltips  and  hover  instructions  when  doing  certain 
interactions,  rather  than  large  chunk  of  text  before  and  after  parts  of  tutorial. 

Monster 

Proof 

1.  The  difference  between  the  first  proof  and  later  equations  scales  rapidly  with  little  opportunity  for  confidence 
building. 

2.  Tutorials  require  too  much  time  -  requires  a  lot  of  memory  retention. 

3.  It  is  very  important  to  give  feedback  when  proper  balance  of  monsters  is  initiated.  Don’t  wait  for  the  player  to 
test  the  equation. 

4.  The  map  felt  too  rigid  for  some  players,  but  the  collection  of  tools  and  early  challenges  excited  many. 

5.  Tutorials  escalate  in  difficulty  too  quickly.  Use  easy  levels  over  and  over  to  build  confidence,  gain  some  tools, 
and  introduce  the  other  game  mechanics  so  players  don’t  lose  faith. 
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Game 

Key  Findings 

Ghost  Map: 
Hyperspace 

1.  Introductory  voiceover  should  contain  a  video  of  some  sort  to  excite  and  engage  players.  The  voiceover  is 
easy  to  click  through  and  miss. 

2.  Greatest  confusion  is  around  Rift  Sealing.  Why  must  they  use  the  bottom  right  sliders  rather  than  interacting 
in  the  primary  map  area? 

3.  Participants  did  not  seem  to  grasp  the  connection  between  Combat  and  Rift  Sealing.  How  does  a  good  seal 
relate  to  killing  the  bugs  and  locking? 

4.  Locking  a  map  in  and  of  itself  is  not  made  entirely  clear.  Players  achieved  this,  but  is  it  just  to  keep  the  map 
clean?  Or  does  it  have  a  larger  purpose? 

5.  Feedback  for  bug  health,  along  with  health  vials  on  the  left  side  of  the  game,  did  not  have  clear  enough 
feedback/values. 

Paradox 

1.  Tutorial  instructions  too  distant  from  paintbrush  actions  and  paintbrush  type;  it  was  difficult  for  players  to 
recall  the  strengths  of  paintbrushes. 

2.  When  new  paintbrushes  are  introduced,  animation  demo  of  how  they  differ  would  be  helpful,  as  would  the 
ability  to  constantly  be  reminded  via  hover  or  by  including  the  name  of  brush  next  to  the  brush  image. 

3.  The  design  palette  is  slightly  soothing  and  a  bit  polarizing;  it  was  too  dull  for  some  players.  But  for  several 
players,  it  achieved  a  nice  tone. 

4.  Scale  and  scope  of  map  size  is  still  a  bit  jarring  when  later  levels  are  played.  Playing  a  map  incrementally, 
starting  from  further  zoomed  in  on  the  difficult  areas,  may  lead  to  a  better  engagement  curve. 

Binary 

Fission 

1.  Simplicity  of  the  game  leads  to  fairly  successful  ramp-up,  but  a  video  demonstration  or  light  back-story  would 
increase  engagement. 

2.  Minimal  feedback  during  filter  splitting  leads  to  doubt;  utilize  sound  or  color  feedback  to  increase  awareness 
of  success  or  failure 

3.  Once  the  levels  get  tough,  understanding  the  mechanics  builds  confidence,  but  -50%  of  the  participants 
wanted  something  more  to  work  towards.  Some  assumed  that  the  correct  filter  path  created  a  larger  pattern 
or  form,  but  this  was  not  ever  confirmed. 

4.  Design  mostly  pleases.  Largest  gap  in  elements  is  how  score  is  calculated. 

5.  Increase  the  competitive  nature  through  friend  referral  and  1-on-l  competition  of  the  same  levels. 

6.  Most  participants  could  have  used  a  slightly  longer  transition  to  harder  levels  while  mechanics  are  given 
feedback  and  the  score  is  explained. 

The  independent  user  feedback  provided  in  these  studies  helped  the  TA1  teams  further 
improve  their  games. 

3.5.3  Computational  Analysis 

We  conducted  a  computational  analysis  of  gameplay  for  the  Phase  2  games  to  better 
understand  the  statistics  surrounding  player  attrition  and  marketing  efforts.  We  developed  a  basic 
framework  for  cleaning  the  data  files  provided  by  TA1  teams  (e.g.,  removing  testing  instances; 
removing  overly  long  play  sessions;  detennining  the  number  of  sessions  played;  detennining 
play  time;  detennining  if  the  player  is  new  or  retained),  and  performing  basic  clustering, 
graphing,  and  statistical  analyses  on  that  data. 

To  determine  which  game  features  lead  to  increased  or  reduced  retention  and/or  productivity, 
we  used  statistical  techniques  to  analyze  the  relationships  between  game  characteristics  and 
game  play  for  the  games  that  provided  significant  data  sets  (Monster  Proof,  Dynamkr,  and 
Paradox).  For  Monster  Proof,  there  was  a  clear  performance  and  retention  difference  between 
players  using  tools,  and  those  not  using  tools;  players  who  did  not  use  tools  tended  to  struggle 
with  the  game,  quit  levels,  and  ultimately  leave.  For  Dynamakr,  players  were  likely  to  leave  if 
they  got  an  initial  level  that  was  too  difficult.  For  Paradox,  players  who  used  the  optimizer  were 
more  successful  and  more  likely  to  be  retained.  Table  2  provides  more  details  on  these  results. 
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Table  2  Statistical  Analysis  Conclusions  for  Monster  Proof,  Dynamakr,  and  Paradox 


Game 

Statistical  Analysis  Conclusions 

Monster  Proof 

■  Players  do  not  seem  to  be  making  good  use  of  the  tools  provided  (e.g.,  infrequent  use  of  free  actions, 
downward  trend  in  using  new  tools). 

■  Tools  are  generally  used  more  when  they  are  first  introduced  (e.g.,  mandated  by  the  training  tools). 

■  Players  are  generally  not  improving  in  the  levels  completed  per  session,  implying  that  this  is  not  a 
choice  made  for  efficiency. 

Dynamakr 

■  Attrition  is  more  likely  on  low  difficulty  and/or  lower  impact  levels. 

■  Attrition  is  more  likely  when  player  performance  is  low. 

■  Attrition  increases  significantly  when  play  count  surpasses  50.  This  implies  that  we  may  want  to 
introduce  rewards  at  regular  intervals  to  overcome  these  attrition  issues. 

Paradox 

■  Retained  players  tend  to  use  the  optimizer  a  lot  more  than  those  leaving  the  game;  it  seems  that  the 
more  players  learn  to  use  the  optimizer,  the  more  likely  they  are  to  keep  playing. 

■  Optimizer  use  also  results  in  higher  scores  in  general. 

3.5.4  Guru  Knowledge  Elicitation 

Working  with  SRI  and  BBN,  we  participated  and  helped  to  manage  two  Guru  events.  The 
first  was  held  on  August  20,  2015,  and  the  second  was  held  on  September  24,  2015.  Players  were 
gurus  in  Ghost  Map:  Hyperspace  and  Binary  Fission. 

We  discussed  the  games  extensively  with  the  guru’s,  and  received  feedback  to  guide  future 
development  efforts.  Some  specific  feedback  included: 

•  There  should  have  been  a  larger  focus  on  community.  For  any  sort  of  public  game,  it  is  essential 
to  set  up  a  thriving  community  that  encourages  collaboration,  socialization,  and  discussion. 
The  community  forums  on  Verigames  did  not  effectively  achieve  that  for  a  number  of  reasons, 
such  as  ease  of  use,  unclear  links  and  proximity  to  actual  games,  lack  of  community 
moderation.  The  Binary  Fission  chat  feature  did  a  better  job;  something  like  that  would  have 
been  great  in  all  of  the  games. 

•  We  should  have  exploited  streaming  from  game  players.  Our  gurus  tended  to  stream  their 
gameplay.  We  could  have  taken  advantage  of  that,  and  linked  those  streams  on  the  Verigames 
website  or  forum  to  show  other  players  how  to  play  effectively. 

•  There  was  a  strong  interest  in  active  participation  in  beta  testing.  A  lot  of  early  adopters  are 
very  interested  in  helping  in  the  debugging  process.  Embracing  that  level  of  feedback  is 
essential  to  testing  and  improving  the  game  system. 

The  guru’s  provided  a  deep  evaluation  of  the  latest  update  to  BBN’s  Hvperspace  game.  In 
addition,  they  gave  an  initial  evaluation  of  Dynamakr,  Binary  Fission,  and  Paradox,  providing 
generally  useful  feedback  for  each  of  those  games.  More  time  was  spent  on  Dynamakr  and 
Binary  Fission,  in  particular,  as  the  developers  for  those  games  participated  via  web  conference. 
BBN  recorded  a  video  of  the  guru’s  Paradox  play,  and  provided  this  video  to  University  of 
Washington  to  provide  them  feedback  for  moving  forward. 

Concluding  our  guru-based  analysis,  we  found  that  gurus  provide  useful  feedback  for 
improving  the  games  and  addressing  potential  game  issues.  Both  of  our  gurus  focused  on 
Hyperspace.  For  those  games,  our  gurus  tended  to  achieve  success  primarily  because  they 
enjoyed  the  game  enough  to  continue  playing  at  a  rate  that  contributed  useful  formal  verification 
content.  They  were  not  able  to  discern  any  specific  strategies  that  seemed  conducive  to  game 
success  (indeed,  in  some  cases,  they  actually  were  using  strategies  that  did  not  benefit  them  in 
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any  way  whatsoever).  When  observing  their  game  play,  we  did  not  discern  useful  strategies  for 
training  or  automation  purposes.  While  it  may  be  the  case  that  gurus  for  other  games  may  have 
provided  more  clear  benefits,  because  they  did  not  participate  in  guru  events,  we  were  unable  to 
interview  them. 

However,  gurus  did  identify  issues  in  existing  training  materials  (e.g.,  points  where  the 
training  materials  were  confusing  or  misleading),  and  identified  game  issues  and  bugs.  We 
observed  that  direct  discussion  between  the  developers  and  the  guru  provided  both  parties  with 
great  insights  into  the  game,  allowing  developers  to  improve  their  game  mechanics  and 
objectives,  and  gurus  to  improve  their  approach  to  playing  the  game.  We  would  certainly 
recommend  that  future  efforts  maintain  an  active  community  interaction  between  developers  and 
game  players  within  a  beta-test  fonnat,  allowing  game  players  to  interact  directly  with 
developers  to  provide  feedback  and  discuss  issues.  We  expect  that  this  strategy  will  both  identify 
problems  in  the  game  and  lead  to  critical  insights  into  the  ways  that  players  approach  these 
games.  These  insights  will  lead  to  the  design  of  better  tutorials,  better  game  mechanics,  and 
ultimately  better  citizen  science  tools. 

3.5.5  WebEx  Event  for  Contest  Winners 

We  held  the  WebEx  event  with  contest  winners  in  October,  and  explored  questions  of  game 
strategy  with  them.  As  we  anticipated,  our  results  were  similar  to  other  games — that  is,  the 
players  had  more  success  through  brute  force  than  through  any  discemable  strategy  that  could  be 
taught  or  automated. 

3.6  Develop,  Manage,  and  Assess  Alternative  Crowd  Sourcing  Incentives 

To  explore  the  success  of  other  incentive  methods  for  crowd  sourced  formal  verification,  we 
assessed  pay-for-play  programs.  In  particular,  our  objective  was  to  compare  the  cost  of 
contributions  from  pay-for-play  methods  compared  to  the  costs  of  similar  contributions  from 
pure  game-based  approaches  and  from  current  standard  methods  (e.g.,  fonnal  verification 
companies).  Specifically,  we  explored  two  pay-for-play  systems.  Amazon’s  Mechanical  Turk,  a 
payment-based  crowd  sourcing  mechanism,  and  TopCoder  Bug  Hunts,  in  which  TopCoder 
community  members  were  paid  to  find  bugs  in  active  game  levels  for  each  game.  We  also  held  a 
contest  in  which  the  best  perfonners  (i.e.,  highest  US  citizen  scorer  at  each  game)  were  to  be 
hired  as  paid  subject  matter  experts  to  provide  insights  into  game  improvements. 

3.6.1  Crowdsourced  Formal  Verification  Using  Amazon’s  Mechanical  Turk 

Amazon’s  Mechanical  Turk  is  a  payment-based  crowd  sourcing  mechanism.  Mechanical 
Turk  organizes  tasks  into  Human  Intelligence  Tasks  (HITs)  that  are  posted  to  the  work  page  and 
perfonned  by  users.  We  organized  HITs  into  levels  for  a  specific  game,  and  they  were  addressed 
on  a  first-come,  first-served  basis.  The  system  was  organized  with  the  following  assumptions: 

•  TA1  games  were  reskinned  to  limit  play  to  specific  levels 

•  The  reskinned  levels  had  the  same  progression  as  the  games 

•  HITs  were  collections  of  some  number  of  levels  with  similar  skill  requirements 

•  TAls  assigned  levels  to  specific  HITs 

•  HITs  associated  with  game  levels  that  are  more  difficult  to  play  were  assigned  higher  prize 
values  than  HITs  associated  with  game  levels  that  are  easier  to  play 
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TopCoder  designed  and  implemented  a  system  for  posting  HITs  to  Mechanical  Turk  and 
synchronizing  task  completion  with  Verigames.com.  In  this  system: 

•  Unique  username  and  password  combinations  were  created  on  verigames.com 

•  Combinations  were  assigned  to  each  HIT 

•  Upon  first  time  use  of  a  combination,  the  user  was  required  to  change  the  password,  thereby 
personalizing  the  associated  HIT  and  preventing  other  users  from  executing  the  same  task 

•  The  AMT  API  was  used  to  synchronize  data  between  Verigames.com  and  AMT 

•  We  budgeted  pay-for-play  awards  beginning  in  June  2015  and  ending  in  August  2015 
We  implemented  the  APIs  for  supporting  Mechanical  Turk  integration  using  Appirio’s 
TopCoder  service  for  the  three  TA1  teams  that  participated  in  the  paid  crowd  sourcing  evaluation 
(Dynamkr,  GhostMap,  and  Paradox). 

When  we  executed  the  Amazon  Mechanical  Turk  (AMT)  experiment,  it  was  a  resounding 
success.  TA1  teams  completed  more  work  in  shorter  time  periods  with  larger  and  more  diverse 
crowds  than  they  had  achieved  with  the  games.  The  cost  of  this  effort  was  minimal  for  the  three 
teams.  The  specific  results  of  these  AMT  experiments  are  fully  addressed  in  the  final  reports  for 
Kestral,  University  of  Washington,  and  Raytheon  BBN.  In  future  programs,  we  strongly 
recommend  exploring  both  a  payment-based  crowd  sourcing  approach  and  a  game-based  crowd 
sourcing  approach.  In  particular,  starting  with  a  payment-based  approach  that  is  less  expensive  to 
initialize  can  enable  a  program  to  provide  immediate  benefit,  and  can  provide  more  time  to 
develop  effective  game-based  techniques  for  more  long-term  engagement  of  citizen  science 
contributors. 

3.6.2  Crowdsourced  Formal  Verification  Using  TopCoder  Bug  Hunt 

We  perfonned  an  additional  investigation  of  paid  crowd  sourcing  using  a  TopCoder  Bug  Hunt. 
For  each  of  the  five  games,  we  ran  a  Bug  Hunt  Event  in  which  TopCoder  community  members 
were  paid  to  find  bugs  in  active  game  levels  for  each  game.  Players  were  only  paid  for  finding 
bugs  in  actual  game  levels,  not  in  tutorials,  thereby  ensuring  that  they  would  work  against  the  FV 
task  while  finding  those  bugs. 

Table  3  shows  the  results  of  these  Bug  Hunts,  both  in  terms  of  the  game  progress  achieved  by 
the  TopCoder  community,  and  in  terms  of  the  number  of  verified  issues  found  in  the  games 
(while  the  latter  was  not  a  primary  objective  of  this  study,  if  it  had  been  done  earlier  in  the 
program,  it  would  have  been  a  very  useful  outcome). 


Table  3  Bug  Hunt  Results 


Binary  Fission 

Dynamakr 

Hyperspace 

Monster  Proof 

Paradox 

Progress 

Levels  Completed: 
332 

Score  Generated: 
21,404,539 

Levels  Completed: 
79 

Levels  Completed: 
4,503 

Levels 

Completed: 

1660 

Valid  Issues 
Reported 

20 

47 

25 

65 

36 
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This  approach  resulted  in  significant  work  achieved,  but  at  a  higher  cost  than  the  Mechanical 
Turk  approach.  In  addition  to  achieving  this  work,  the  TopCoder  community  identified  numerous 
errors  (on  average,  38  per  game)  not  yet  identified  in  previous  testing  and  evaluation  by  game 
players  or  AMT  participants.  This  added  value  can  be  particularly  useful  in  the  early  part  of  a 
release  period,  because  it  allows  game  developers  to  improve  the  software  to  engender  more 
gameplay  in  the  future. 

3.6.3  increasing  Participation  with  Contests  and  Marketing 

Finally,  we  explored  a  marketing  and  reward-based  approach  to  incentivize  game  play  and 
perfonnance.  In  particular,  several  weeks  after  the  Phase  2  game  release,  we  ran  a  community 
contest  that  provided  the  best  performers  with  in-game  rewards  and  an  opportunity  to  participate 
in  a  webinar  with  the  game  developers.  We  gained  useful  feedback  from  these  performers  and 
evaluated  whether  the  marketing  effort  that  identifies  this  incentive  mechanism  increased  overall 
participation  and  outcome  from  the  game  community. 

We  generated  rules  for  this  contest  (see  Appendix  H)  and  generated  a  public  announcement, 
all  approved  by  DARPA.  While  we  originally  intended  to  have  travel-based  prizes  associated 
with  this  contest,  we  detennined  that  these  prizes  were  too  difficult  to  manage  in  the  current 
funding  format.  Instead,  we  rewarded  perfonners  with  game-  and  website-based  accolades  and, 
for  the  best  performers,  participation  in  a  webinar  with  the  TA1  developers.  We  started  this 
contest  in  late- July,  and  executed  it  for  two  weeks  to  evaluate  its  impact  on  participation  within 
the  Verigames  website. 

When  we  executed  the  contest,  it  had  a  clear  impact  on  people  logging  into  the  website. 
However,  for  some  of  the  individual  games,  this  impact  did  not  transition  to  actual  gameplay  (for 
example,  Ghost  Map  Hyperspace  and  Dynamakr  only  had  one  and  two  players  achieve  progress 
during  the  contest,  respectively).  The  most  successful  elements  of  the  contest  were  with  Binary 
Fission  and  Monster  Proof,  where  the  contest  winners  interacted  heavily  with  the  game 
developers  to  ask  questions  and  provide  feedback,  ultimately  helping  to  improve  the  games.  In 
future  efforts,  we  recommend  coupling  contests  with  developer  interaction  to  engender 
participation  and  improve  the  outcome  of  the  contest. 

In  October,  we  held  the  Contest  Winner  WebEx  Event,  the  prize  for  the  winners  of  the 
contest;  the  results  of  this  event  are  discussed  in  Section  3.5.5. 

3.7  Market  CSFV 

We  continued  the  marketing  efforts  started  in  the  earlier  segment  of  the  CSFV  effort, 
focusing  on  a  combination  of  website  branding,  public  relations,  and  marketing  events  designed 
to  attract  attention  to  CSFV  crowd  sourcing  methodologies.  This  effort  was  led  by  GameDocs 
and  Ms.  Belinda  Van  Sickle.  The  main  focus  of  these  efforts  was  on  marketing: 

•  Phase  2  games 

•  Mechanical  Turk  tasking 

•  Perfonnance-based  prize  opportunities 

Table  4  lists  the  specific  marketing  strategies  used  for  the  CSFV  effort. 
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Table  4:  Marketing  Strategies 


Marketing  Method 

Description 

Website  Branding 
Strategy 

Suggest  changes  to  the  website  based  on  the  games  as  developed,  including  creating  overall 
themes,  branding  assets,  and  other  components  that  are  more  relevant  to  the  Phase  2  games. 

Public  Relations 

Create  public  relations  materials  including  press  releases,  PR  articles,  social  media  marketing 
messages,  and  email  blasts  for  the  three  elements  identified  above.  PR  staff  includes  Belinda  Van 
Sickle  and  subcontractors  of  GameDocs,  including  successful  game  industry  PR  agencies. 

Media  Buys 

Purchase  online  advertising  on  appropriate  sites  based  on  marketing  strategy  for  each  title.  Online 
advertising  runs  on  major  social  media  sites,  depending  on  the  audience  for  each  title  and  contest. 
Media  buying  staff  include  Belinda  Van  Sickle  and  other  successful  game  industry  PR  agencies. 

Ad  Performance 
Tracking  and 
Optimization 

Track  the  performance  of  online  advertising  on  all  channels  including  analyzing  where  users  are 
coming  from,  what  campaigns  are  the  most  successful,  and  what  campaigns  are  underperforming, 
with  the  goal  of  refining  the  focus  on  effective  campaigns. 

Because  the  TA1  teams  needed  to  focus  on  three  things  for  the  first  few  weeks  of  launch:  bug 
reporting  and  fixing;  usability  testing  infonnation  integration  and  updates;  player  community 
engagement  retention  work,  we  ensured  our  marketing/community  management  strategy  allowed 
the  teams  to  ensure  bugs,  usability,  community,  and  engagement  received  the  necessary 
resources. 

The  following  sections  describe  the  individual  marketing  activities  conducted  during  the 
effort.  Marketing  activities  conducted  for  specific  tasks  are  described  in  those  sections 
throughout  the  report. 

3.7.1  Marketing  Plan 

We  updated  the  original  2015  marketing  plan  to  confonn  to  current  timeline,  budget  and 
game  functionality.  The  marketing  plan  is  detailed  in  Appendix  G.  We  updated  our  plan  to  focus 
on  Citizen  Science  on  verigames.com  as  well  as  calls  to  action.  Once  users  go  to  verigames.com 
from  various  press  and  other  sources,  the  main  call  to  action  is  “Play  Now.”  However,  if  that 
user  is  interested  in  scientific  content,  the  story  behind  the  Veri games  project,  the  science  behind 
DARPA’s  CSFV  program,  etc.,  the  call  to  action  is  to  visit  the  Citizen  Science  pages  and  satisfy 
their  interest  through  the  various  content  found  there.  The  call  to  action  on  the  Citizen  Science 
section  is  “Play  Now.”  Once  users  have  satisfied  their  scientific  interest  in  the  Verigames 
program  and  DARPA’s  CSFV  project,  the  call  to  action  is  to  play  the  games.  We  also  updated 
the  plan  to  focus  on  community  events. 

3.7.2  Community  Management 

We  created  a  strategic  plan  for  Phase  2,  which  included  TA1  participation;  required 
marketing  materials;  social  media  plan;  verigames.com  plan;  public  relations  (PR)  integration; 
online  advertising  integration;  acquisition,  conversion  and  retention  plan;  and  viral  marketing. 

We  received  DARPA  Public  Affairs  Office  (PAO)  approval  for  the  community  events  schedule. 

We  conducted  these  events  for  the  TA1  teams,  including  a  live  in-game  chat,  a  video  series, 
and  live  online  chats.  Three  teams’  community  events  were  archived  on  Verigames’  online  social 
media.  Community  events  implementation  involved  extensive  technical  testing  and  the  creation 
of  instructional  materials  for  TA1  teams  and  Verigames  users.  We  used  Google  Hangouts  and 
Google  Hangouts  on  Air  technology  to  allow  for  live  chats  and  live  chats  with  unlimited  users. 
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These  chats  are  not  familiar  to  everyone  and  required  testing  and  instructions  to  ensure  they 
would  perform  properly  during  scheduled  events.  The  testing  and  instructions  worked  and  we 
had  no  issues.  The  community  events  worked  well  and  the  archived  versions  are  up  on  the 
Verigames  YouTube  channel.  (Google  Hangouts  technology  offers  automatic  archiving, 
although  we  edited  the  archived  versions  to  remove  blank  air.) 

3.7.3  Google  Adwords  Campaign 

Small  ad  buys  in  fourth  quarter  2014  gained  market  intelligence  on  what  ad  buys  work  for 
gaining  conversion  to  players. 

For  our  comparison  launch  ad  campaign,  we  gathered  information  from  TA1  teams  on  data 
collected  in  Phase  1 .  One  team  had  concrete  information;  one  team  had  conditional  information; 
and  one  team  had  ambiguous  information.  We  compared  Phase  1  collected  data  to  real  world 
costs/scale/free  up  time  of  formal  verifiers.  Because  the  data  was  not  solid,  we  created  a  launch 
campaign  that  focused  on  the  Citizen  Science  aspect  of  Verigames  with  a  minor  comparison 
factor  that  was  deleted  so  that  we  could  receive  approval  from  the  DARPA  Public  Affairs  Office 
(PAO).  Then,  we  implemented  our  Phase  2  Google  Adwords  campaign  with  PAO  approval  of 
ads  and  repeated  revision  of  ads  due  to  some  formatting  issues.  Analytics  showed  a  spike  in 
verigames.com  hits  and  gameplay  and  the  submission  of  solutions. 

Frequent  review  of  Google  Adwords  campaign  analytics,  Logaholics  verigames.com  site 
analytics,  and  in- game  play  stats  from  Binary  Fission,  Monster  Proof  and  Paradox  maintained 
the  efficacy  of  the  program.  Low  performing  ads  were  eliminated  and  global  territories 
continually  optimized  to  get  the  best  perfonnance  from  the  campaign. 

3.7.4  Press  Calendar,  Press  Releases,  and  Press  Kit 

We  created  a  press  calendar  for  launch  press  releases  and  created  press  kits  for  all  five 
games.  We  compared  release  service  costs  and  coverage  from  two  vendors  and  selected  the  best 
vendor  for  the  effort.  After  we  received  PAO  approval,  we  disseminated  press  releases  through  a 
science-related  press  service.  We  uploaded  the  final  press  kit  to  verigames.com  and  made  it 
available  to  press  outlets.  We  updated  the  press  kit  several  times  during  the  effort  with  updated 
press  releases,  updated  links,  and  other  material  as  needed. 

3.7.5  Social  Media 

Social  media  marketing  was  a  secondary  strategy  for  Phase  2.  To  serve  the  citizen  scientist 
market,  the  verigames.com/in-game  sharing  focus  for  Phase  2  was  on  scientific  content.  The 
Citizen  Science  page  content  was  sharable. 

We  performed  social  media  marketing  through  Facebook,  Twitter,  and  Google+.  We 
announced  the  launch,  posted  launch  content,  new  blog  posts  and  other  new  material  through 
these  channels.  We  used  paid  social  media  boosting  on  Facebook  and  Twitter  for  all  community 
events  and  hosted  press  events. 

3.7.6  Newsletters 

We  published  the  March  2015  newsletter  with  previously  unpublished  content.  The  April 
version  consisted  of  a  teaser  with  art  and  logos  of  all  Phase  2  games.  We  sent  a  newsletter  with  a 
launch  announcement  to  all  registered  users  shortly  after  the  DARPA  web  feature  was  released. 
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We  increased  the  frequency  of  the  newsletter  to  provide  a  forum  for  promotion  of  community 
events  and  to  showcase  major  press  events.  Analytics  have  showed  steady  user  engagement  with 
increased  frequency  as  well  as  no  increase  in  negative  feedback  (unsubscribes  or  marks  as 
spam). 

Finally,  we  delivered  a  closing  Thank  You  newsletter  at  the  end  of  November  2015. 

3.7.7  Public  Relations 

We  sent  an  email  to  several  editors/outlets  that  have  covered  crowd  sourced  games  in  the 
past  and  approached  outlets  that  covered  launch,  offering  interviews  for  feature  articles.  We 
received  a  press  inquiry  from  a  BBC  reporter  and  scheduled  interviews  with  TA1  team  members, 
and  provided  press  kit  materials  and  other  background  information  on  the  effort.  We  ensured  all 
development  team  members  were  apprised  with  current  PAO  direction  for  press  contact  before 
their  interviews. 

We  posted  the  published  article  on  verigames.com,  Verigames  social  media,  and  in  the 
newsletter.  The  article  resulted  in  a  major  spike  in  gameplay  and  solutions  submitted  to  the 
games.  We  followed  coverage  of  Verigames  on  online  outlets  throughout  the  effort  and  re-posted 
content  as  appropriate  to  the  website,  newsletter,  and  social  media. 

3.7.8  Clip  Reporting 

We  temporarily  hired  a  clipping  service  to  aggregate  all  Verigames-related  media,  but  after  a 
10-day  trial,  the  results  missed  materials  found  in  dated  Google  searches,  so  we  did  not  engage 
this  service. 

3.7.9  Contest  Materials 

We  created  promotional  content  for  the  community  contest  on  the  website  (News  page  and 
drop-down  banners)  and  implemented  paid  social  media  marketing,  verigames.com  promotions, 
and  newsletter  promotions.  We  ensured  that  the  rules  and  other  necessary  content  were  on 
verigames.com  to  support  the  content. 

After  the  contest,  winner  copy  was  created  and  delivered  to  the  contest  winners.  We 
announced  the  winners  in  the  newsletter  and  on  social  media,  and  posted  announcements  to 
verigame.com. 

3.8  Manage  Program 

We  managed  the  program  and  guided  the  teams  through  a  smooth  public  launch,  multiple 
contests  and  events,  and  maintained  the  website  through  November  30,  2015,  at  which  time  we 
removed  the  games  and  created  a  Verigames.com  website  with  links  to  TAl-hosted  websites  for 
their  individual  games. 

Charles  River  Analytics  understands  that  structured,  open,  collaborative  management  is 
critical  to  system  integration  success  in  large  research  and  development  efforts;  therefore,  we 
focused  on  early  and  continuous  coordination  and  collaboration  with  the  DARPA  PM, 
Government  representatives,  and  other  CSFV  perfonners.  Our  team  held  weekly  coordination 
meetings  and  encouraged  other  perfonners  to  test  the  framework  and  integrate  their  components, 
enabling  our  team  to  solve  problems  in  real  time.  This  incremental  testing  greatly  improved  the 
probability  of  success  and  reduced  schedule  risk  for  all  performers  during  test  phases.  Charles 
River  also  followed  a  disciplined  Program  Management  process  based  on  the  Project 
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Management  Institute’s  Project  Management  Body  of  Knowledge  (PMBOK  v4),  modified  for 
the  unique  flexibility  demanded  of  successful  R&D  programs.  Our  process  focused  on 
continuous  Quality  Assurance  and  Control,  careful  Configuration  Management,  and 
comprehensive  Risk  Management;  areas  that  routinely  drive  successful  program  results. 

Our  team  was  led  by  Charles  River  Analytics  Inc.,  who  led  the  analytics  effort  and  managed 
both  the  maintenance  and  update  effort  and  the  paid  crowd  sourcing  research.  Specifically,  Mr. 
Sean  Guarino,  Principal  Scientist,  was  the  Principal  Investigator  for  this  effort,  bringing  his 
extensive  previous  cognitive  systems  engineering  (CSE)  and  game  design  experience  to  guide 
the  assessment  of  current  games,  and  the  evaluation  of  expert  (guru)  play  styles.  Mr.  William 
Dorin,  Software  Engineer,  was  the  Technical  Lead  for  this  effort,  guiding  the  development 
process,  and  performing  maintenance  and  engineering  tasks  where  necessary  to  augment  crowd 
sourced  development. 

Our  team  maintained  the  same  team  members  from  the  previous  effort  to  work  as 
subcontractors  on  this  effort.  Appirio  joined  us  to  lead  the  continuing  development  effort  of  the 
Verigames  website  and  the  integration  framework.  Specifically,  Appirio  continued  to  run 
contests  using  the  TopCoder  community  to  achieve  maintenance  and  development  tasks, 
augmenting  the  framework  to  incorporate  both  Phase  2  games  and  task-oriented  approaches  for 
Mechanical  Turk.  By  maintaining  Appirio  in  this  role,  we  continued  to  have  access  to  Mr.  Ambi 
Del  Villar  and  Mr.  Ahmad  Alkhawaja,  who  are  both  intricately  familiar  with  the  existing  code 
base.  Our  team  was  also  joined  by  GameDocs,  led  by  Ms.  Belinda  Van  Sickle,  who  continued  to 
lead  the  marketing  effort  for  CSFV.  Supporting  Ms.  Van  Sickle,  we  continued  to  include 
YouEye  on  our  team  to  gather  and  run  users  for  evaluations  of  CSFV.  Our  team  tactically  used 
other  companies  who  reviewed  Verigames  in  Phase  I,  including  WhiteHat  Security  to  evaluate 
the  security  of  the  updated  Verigames  site. 

The  organization  of  our  team  is  shown  in  Figure  5. 


Charles  River  Analytics 

Management  Team 

Dr.  Ryan  Kilgore,  PM 
Mr.  Don  Reynolds,  Finance  &  Contract  Lead 
Mr.  Frank  Veronesi,  Financial  Tracking 
Mr.  Mark  Felix,  Contract  Management 
Ms.  Becky  Afergan,  Reporting  &  Deliverables 
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Figure  5  Program  Organization 


Technical  Team 

Mr.  Sean  Guarino,  PI 
Mr.  William  Dorin,  Technical  Lead 
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Charles  River’s  process  for  achieving  program  cost,  schedule,  and  performance  goals  is 
based  on  a  practical  and  proven  process  that  focuses  on  responsiveness  and  accurate  infonnation 
products  for  our  Program  Managers.  This  process,  shown  in  Figure  6,  works  particularly  well  for 
integrating  scientific  and  R&D  programs  that  require  management  of  frequently  changing 
requirements. 


Corrective  Action 
Stage: 


Determine  Solution 
Implement  Solution 
Assess  Results 


Job  Cost 
Information 


Figure  6  Charles  River’s  Cost,  Schedule,  and  Performance  Management  Process 
Our  process  consists  of:  the  following  three  stages: 

•  Planning  Stage  -  Determines  the  cost,  schedule,  and  performance  requirements  and  the  metrics 
for  monitoring  progress.  The  Planning  Stage  identifies  project  risks  and  mitigation  approaches; 
suitable  project  metrics,  resources,  and  the  budget;  schedule  milestones,  and  products  for 
operational  tasks. 

•  Overview  Stage  -  Monitors  progress  against  the  metrics.  In  the  Overview  Stage,  we  analyze 
results  and  progress  indicators  from  the  operational  tasks,  which  include  performing  the  work, 
reviewing  the  products,  and  producing  the  deliverables.  We  also  produce  and  monitor  biweekly 
budget  data. 

•  Corrective  Action  Stage  -  Recovers  cost,  schedule,  and  performance  compliance  when 
problems  arise.  The  key  to  achieving  this  compliance  is  early  detection  of  problems  and  quick 
action,  enabled  by  our  agile  software  engineering  process.  In  the  Corrective  Action  Stage,  the 
PM  works  in  concert  with  team  members  and  the  customer  to  determine  and  implement  the 
best  solution  for  recovering  from  a  cost,  schedule,  or  performance  deviation. 

This  process  is  effective,  responsive,  predictive,  and  our  experience  has  demonstrated  it 
responds  quickly  to  changing  requirements. 

Because  R&D  programs  are  risky,  the  Program  Manager  must  implement  a  robust,  structured 
risk  management  process  to  ensure  program  success  despite  uncertainties.  Charles  River 
aggressively  managed,  mitigated,  and  reduced  risk  throughout  the  life  of  the  program.  This 
process  began  during  proposal  development  and  continued  through  the  program  as  potential  risks 
were  identified,  analyzed,  and  classified.  Risk  mitigation  and  reduction  strategies  reduced  the 
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probability  of  certain  risks  occurring.  Our  team  also  developed  strategies  for  any  risks  that 
threatened  program  success  (see  table  5). 

Table  5  Program  Management  Risks  and  Mitigation  Plans 


Management  Plan  Risk 

Mitigation  Plan 

Inter-team  communications 

Our  subcontractor  and  consultant  had  clearly  stated  tasks  focused  on  regular  reporting 
and  interaction  with  Charles  River  as  the  Prime  Contractor;  all  parties  agreed  to  attend 
team  meetings  and  support  team  events  as  well  as  Program  events. 

Interactions  with  other  CSFV 
performers 

Regular  Technical  Interchange  Meetings  (TIM)  with  all  performers  clearly  established 
interface  standards  early  and  regularly.  Our  agile  engineering  process  produced  informal 
software  drops  every  two  weeks  so  that  other  performers  could  identify  areas  of  potential 
concern  without  waiting  for  formal  code  drops. 

Delayed  component  releases 
from  TA1/TA2 

Our  agile  engineering  approach  encouraged  regular  releases  of  components  from  CSFV 
performers.  All  performers  worked  with  Charles  River  to  ensure  that  components  were 
delivered  on  time  and  that  any  obstacles  to  those  releases  are  cleared  efficiently. 

Execution  of  commercial 
contracts 

We  reviewed  contract  needs  with  Appirio  and  other  commercial  contractors  in  advance, 
and  were  comfortable  that  we  could  negotiate  a  commercial  contract  with  the  company; 
however,  there  was  a  small  risk  that  there  would  be  necessary  flow  down  clauses  from  the 
AFRL  contract  that  proved  unacceptable  to  Appirio.  Should  those  issues  have  proved 
unsurmountable,  we  were  prepared  to  bring  on  Mr.  Ahmad  Alkhawaja  as  a  direct 
consultant  to  transition  ownership  of  the  software  to  the  Charles  River  engineering  staff. 

Because  risk  management  is  not  a  onetime  event,  Charles  River  Analytics’  program  teams 
continuously  monitored  program  risk  at  our  internal  monthly  program  reviews  to  reevaluate  the 
status  of  risks  tracked  on  the  Risk  Register  and  identify  any  new  risks  that  occurred  as  a  result  of 
work  during  the  previous  period.  All  team  members  were  encouraged  to  identify  risks  to  the 
Program  Manager  and  Principal  Investigator  at  any  time. 
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4.  SUMMARY  OF  RESULTS 


Overall,  the  CSFV-A  program  was  a  strong  success.  Working  with  the  TA1  game 
development  teams,  we  successfully  managed  the  release  of  the  Phase  2  games,  performed 
analysis  of  those  games,  and  assessed  additional  incentive  mechanisms  for  future  crowdsourcing 
efforts. 

Our  primary  focus  during  the  effort  was  to  manage  the  Verigames  website  and  integration 
framework.  In  support  of  this,  we  updated  the  website  to  enable  the  integration  of  the  Phase  2 
games,  and  managed  both  the  initial  friends  &  family  release  of  the  games,  and  the  public  release 
of  those  games.  In  updating  the  website,  we  built  a  Citizen  Science  page  that  helps  to  reframe 
CSFV  as  a  citizen  science  effort,  rather  than  a  gaming  effort,  providing  a  variety  of  infonnation 
to  the  community  on  the  purpose  and  underlying  technology  for  CSFV.  Throughout  the  effort, 
we  addressed  software  and  integration  issues  that  were  identified  by  the  TA1  teams  in  a  timely 
manner,  and  ensured  that  the  website  was  consistently  maintained  in  this  period.  At  the  close  of 
the  effort,  we  transitioned  the  website  to  a  simpler,  static  version  of  the  website  that  maintains 
the  citizen  science  information,  but  primarily  provides  a  long-term  pointer  to  games  that  are 
maintained  by  TA1  teams. 

Another  key  focus  of  our  effort  was  analytics  involving  the  CSFV  games.  We  primarily 
focused  on  two  types  of  analytics  that  were  identified  as  most  useful  by  the  TA1  teams:  (1) 
expert  game  design  analysis  and  (2)  user  studies.  We  performed  deep  reviews  of  the  games  with 
our  expert  game  designers,  providing  early  feedback  to  the  teams  to  help  improve  the  games. 
Then,  for  much  of  the  effort,  we  held  biweekly  game  design  meetings  with  the  majority  of  the 
TA1  teams  to  address  common  issues  across  the  games.  Also,  during  the  friends  &  family 
release  period,  we  worked  with  YouEye  to  perform  a  user  analysis  of  the  games,  gathering  key 
user  feedback  on  pain  points  in  the  game.  Each  of  these  techniques  were  identified  as  very  useful 
by  the  TA1  teams  to  drive  their  game  design  and  refinement. 

Other  analytical  efforts  were  less  useful.  In  particular,  we  did  manage  interactions  with  game 
gurus,  and  acquired  some  useful  feedback  from  those  gurus.  However,  while  game  gurus  did 
perform  well  on  their  games,  all  indications  were  that  this  was  based  primarily  on  time 
commitment,  not  on  any  kind  of  a  discernable  or  useful  game  strategy.  Therefore,  we  were  not 
able  to  make  valid  recommendations  for  tutorial  improvements  or  automation,  beyond 
addressing  guru  issues  with  tutorial  clarity  and  usefulness.  Finally,  our  computational  analytical 
efforts  also  provided  minimal  benefit;  we  identified  several  important  trends  in  play  style,  but 
these  did  not  generate  recommendations  for  game  improvements.  In  general,  this  effort  was 
impacted  by  the  same  issues  we  had  in  our  previous  work — the  ability  to  analyze  play  patterns 
was  strongly  limited  by  the  limited  number  of  repeat  players  across  the  games. 

Another  key  objective  of  our  effort  was  to  enable  and  help  to  assess  alternative  incentive 
mechanisms  for  driving  crowdsourced  formal  verification.  Largest  amongst  these  efforts  was  our 
work  supporting  the  TA1  teams  with  an  integration  framework  for  using  Amazon’s  Mechanical 
Turk  to  acquire  participants  and  address  fonnal  verification  problems.  The  use  of  AMT  was  a 
resounding  success,  with  results  that  in  many  ways  surpassed  those  of  the  game-based 
crowdsourcing.  The  details  of  these  results  are  captured  in  the  relevant  TA1  final  reports; 
however,  we  strongly  recommend  using  AMT-based  crowdsourcing  in  future  citizen  science 
efforts  as  an  early  and  inexpensive  method  to  engender  participation. 
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We  also  explored  TopCoder  Bug  Hunts  as  another  payment-based  approach  to  engender 
participation.  This  was  also  a  success,  as  members  of  the  TopCoder  community  solved  a  large 
number  of  puzzles  in  the  games  as  they  tried  to  address  bugs.  However,  this  is  dramatically  more 
expensive  than  the  AMT-based  approach.  Still,  early  on  in  a  game-based  effort,  this  can  be  very 
useful,  as  it  does  not  only  address  the  underlying  citizen  science  objectives,  but  also  provides 
numerous  bug  reports  that  game  developers  can  address  to  ensure  that  the  game  is  more  stable 
and  engaging  for  free  participants. 

Finally,  we  explored  a  contest-based  incentive  mechanism  that  was  ultimately  unsuccessful, 
as  it  did  not  drive  increased  participation  for  the  games.  We  were  not  really  able  to  run  the 
contests  that  we  had  intended  due  to  payment  restrictions  on  Government  contract  vehicles;  in 
the  long  term,  we  recommend  refining  these  kinds  of  contests  to  incorporate  more  substantial 
rewards  from  an  early  point  in  the  effort,  and  further  test  if  they  can  be  useful  in  garni  lied 
crowdsourcing  efforts. 

In  addition  to  these  primary  technical  thrusts,  throughout  the  effort,  we  successfully  managed 
cross-team  collaboration,  successfully  marketed  CSFV  with  the  aid  of  GameDocs,  and  managed 
the  execution  of  the  CSFV  integration  effort. 
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5.  CONCLUSIONS  AND  RECOMMENDATIONS 


Overall,  we  are  pleased  with  the  success  of  our  integration  effort  in  the  CSFV  program.  We 
successfully  updated  and  maintained  the  verigames.com  website  to  enable  the  deployment  of  the 
Phase  2  games,  and  the  collection  of  player-generated  solutions  and  performance  data  on  those 
games.  Overall,  releases  were  done  in  a  timely  manner  with  respect  to  the  targeted  schedule,  and 
the  website  was  stable  through  the  release  of  those  games.  Further,  the  software  we  built  for  that 
distribution  provides  an  environment  that  could  potentially  be  reused  in  future  applications, 
providing  a  basic  carousel  format  that  enables  the  simple  presentation  of  a  range  of  citizen 
science  solutions. 

We  are  also  particularly  pleased  with  the  results  of  our  key  expert  and  user  analytics  in 
evaluating  the  five  games  developed  by  TA1  teams.  Across  the  CSFV  effort,  we  were  able  to 
make  significant  recommendations  that  contributed  to  game  improvement,  helping  the  game 
designers  to  find  and  address  key  flaws  in  their  games.  For  future  game-based  citizen  science 
efforts,  we  would  recommend  similar  analysis  efforts.  By  reviewing  games  with  expert  game 
developers,  and  holding  cross-team  game  design  meetings,  the  overall  development  community 
is  able  to  generate  better  games,  improving  engagement,  usability,  and  tutorial  effectiveness. 

User  studies  are  similarly  essential  to  effective  game  development,  providing  critical 
feedback  on  which  elements  players  are  finding  difficult  to  comprehend  or  use.  Indeed,  more 
standard  game  communities  use  beta  releases  of  the  game  as  a  means  to  enable  effective  user 
studies  and  user-driven  improvements  to  games.  For  future  efforts,  we  recommend  supporting  an 
early,  interactive  beta-testing  community  to  evaluate  and  improve  these  games.  Some  of  the 
biggest  improvements  made  to  games  in  CSFV  happened  because  developers  were  directly 
interacting  with  the  TA1  developers;  in  future  citizen  science  efforts,  we  recommend  enabling 
that  kind  of  interaction  across  the  program.  The  community  would  be  better  served  with  an  early 
release  of  game  concepts — even  if  those  concepts  are  not  fully  operational — to  start  getting  user 
feedback  from  an  early  stage.  Furthermore,  involving  users  in  this  way  in  the  community  would 
foster  a  greater  sense  of  community  involvement,  which  in  turn  would  have  the  potential  to 
motivate  further  participation  when  the  full  games  are  released,  and  data  is  being  collected. 

Another  key  success  in  the  effort  was  the  use  of  Amazon’s  Mechanical  Turk  and  TopCoder’s 
Bug  Hunts  to  generate  additional  game  play  and  problem  solving.  Both  payment-based 
approaches  were  very  successful  in  increasing  community  contributions.  AMT,  in  particular, 
generated  better  results  than  the  games,  in  a  shorter  period,  and  for  relatively  low  cost.  The  Bug 
Hunts  were  more  expensive,  but  also  generated  a  large  amount  of  solution  traffic  and  at  the  same 
time  identified  numerous  game  bugs  that  could  be  resolved  to  improve  the  community 
experience.  In  future  efforts,  we  strongly  recommend  using  payment-based  crowdsourcing  as  a 
starting  point,  beginning  the  effort  with  tools  like  AMT,  where  the  initial  focus  is  simply  to 
crowdsource  the  problem,  and  later  introducing  concepts  such  as  gamification,  where  the  focus  is 
to  extend  that  capability  to  a  tool  with  long-tenn  viability  (e.g.,  a  tool  that  citizen  scientists  will 
contribute  to  without  payment). 

We  experienced  several  limitations  with  respect  to  other  elements  of  the  CSFV-A  program. 
One  of  our  key  objectives  was  to  analyze  the  behaviors  of  game  gurus — those  game  experts  who 
were  responsible  for  generating  the  largest  contributions  to  proofs.  We  had  originally  thought 
that  such  experts  might  have  game  strategies  and  approaches  that  would  be  useful  to  share  with 
other  players  or  to  incorporate  into  underlying  automation  for  fonnal  verification.  Ultimately,  we 
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had  two  issues  in  executing  this  research.  First,  due  to  the  constraints  on  the  funding  mechanisms 
for  these  gurus,  we  were  unable  to  find  many  gurus  willing  to  participate  in  an  event.  In  the 
future,  we  recommend  using  more  online  interactions  to  control  these  costs,  as  we  were  more 
successful  in  fostering  participation  in  online  meetings  with  contest  winners.  More  importantly, 
the  gurus  who  contributed  most  effectively  to  these  games  did  not  actually  appear  to  be  using 
strategies  in  the  games;  rather,  they  were  the  individuals  that  played  the  games  the  most. 
Nevertheless,  it  was  useful  to  engage  the  game  gurus,  primarily  because  they  understood  the 
game  experience  and  were  able  to  provide  detailed  and  essential  feedback  that  can  improve  the 
game  experience.  In  future  research,  we  recommend  early  engagement  with  high-volume  players 
on  a  regular  basis  to  understand  what  they  like  and  do  not  like  about  the  game.  These  players  are 
the  real  contributors  in  a  citizen  science  community,  and  game  improvements  should  be  made 
with  their  engagement  as  a  primary  goal. 

With  respect  to  computational  analytics,  the  key  issue  that  we  had  throughout  the  effort  was  a 
lack  of  good  data  from  the  games.  Part  of  this  issue  stemmed  from  limited  collection  and 
delivery  of  data  from  some  of  the  game  environments,  but  most  of  the  issue  arose  from  the  very 
limited  player  population.  Nevertheless,  we  were  able  to  do  some  basic  analyses  of  game 
characteristics  that  seemed  to  be  most  linked  to  attrition,  and  were  able  to  provide  some 
reasonable  data  to  TA1  teams  to  guide  game  improvements.  Overall,  even  when  we  were 
successful  in  identifying  these  patterns,  the  infonnation  provided  was  not  as  useful  to  TA1  teams 
as  information  provided  by  expert  game  designer  analyses  or  user  studies. 

Finally,  we  did  hold  a  single  contest  late  in  the  effort  to  try  to  increase  participation  rates. 
While  it  had  a  minor  effect  on  traffic  on  the  Verigames  website,  its  impact  on  actual  game  play 
was  negligible.  Part  of  this  may  be  because  of  the  limited  prizes  we  were  able  to  provide  (i.e.,  no 
cash  or  material  prizes;  prizes  were  limited  to  accolades  in  the  game  environment  and/or  social 
meetings  with  game  developers).  Nevertheless,  we  believe  that  more  regular  contests  that  are 
well-integrated  into  the  game  environment  and  the  games  themselves  can  provide  a  method  to 
increase  traffic,  and  is  worth  investigating  further  in  future  efforts. 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 


ACRONYM 

DESCRIPTION 

3D 

Three  Dimensional 

GSE 

Gamification  in  Security  Education 

AFRL 

Air  Force  Research  Laboratory 

AMT 

Amazon  Mechanical  Turk 

API 

Application  Programming  Interface 

AWS 

Amazon  Web  Services 

CPU 

Central  Processing  Unit 

CSE 

Cognitive  Systems  Engineering 

CSFV 

Crowd  Sourced  Formal  Verification 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DEPEND 

Detection,  Explanation,  and  Prediction  of  Emerging  Network  Developments 

DNS 

Domain  Name  System 

E&RM 

Exploitation  and  Resource  Management 

EMIAT 

Evolvable  Microgames  for  Information  Assurance  Training 

FV 

Formal  Verification 

GUARD  DOG 

Group  Understanding  &  Analysis  for  Rapid  Detection  -  Deployed  on  Ground 

HIT 

Human  Intelligence  Task 

HMI 

Human  Machine  Interface 

IA 

Information  Assurance 

INTERAACT 

Interface  for  Enhanced  Analyst/Automation  Collaborative  Tasking 

JAGUAR 

Joint  Air/Ground,  Unified,  Adaptive  Replanning 

MAAGI 

Malware  Analysis  &  Attribution  using  Genetic  Information 

MITM 

Man-in-the-Middle 

PAO 

Public  Affairs  Office 

PI 

Principal  Investigator 

PM 

Project  Manager 

PMBOK 

Project  Management  Body  of  Knowledge 

PR 

Public  Relations 

QA 

Quality  Assurance 

QC 

Quality  Control 

SME 

Subject  Matter  Expert 

TIM 

Technical  Interchange  Meetings 

TA1 

Technical  Area  1  of  the  CSFV  Program 

TA2 

Technical  Area  2  of  the  CSFV  Program 

US 

United  States 

VM 

Virtual  Machines 
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Appendix  A.  Integration  Activities 

Table  6  summarizes  the  issues  addressed  as  part  of  the  integration  activities  during  the  effort. 


Table  6  Integration  Issues 


Issue  Type 

Priority 

Status 

Summary 

Helpdesk 

Ticket 

Major 

Open 

After  /verifySession,  what  should  we  do  with  userid? 

Bug 

Minor 

Resolved 

"Cannot  upload  photo”  error  on  verigames.com 

Bug 

Major 

Resolved 

Problem  adding  score  records  to  api.dynamo.verigames.org  -  error  400 

Task 

Major 

Resolved 

Create  Forum  categories  for  Beta  testing 

Task 

Major 

Resolved 

Create  email  blast;  add  lists  on  MailChimp 

Helpdesk 

Ticket 

Critical 

Resolved 

Change  to  binaryfission  domain  name 

Bug 

Major 

Resolved 

Dynamo  -  Can't  get  user  info  using  "Resource  Owner  Credentials  Grant" 

Oauth.  Returns  "not  authorized" 

Bug 

Major 

Resolved 

The  login  page  as  part  of  Oauth  redirect  does  NOT  include  a  “forgot 
password”  link. 

Bug 

Major 

Resolved 

Error  returned  when  calling  Scoring  api  -  400.  The  query  parameter 
"playerlds"  should  be  valid  array  of  strings. 

Bug 

Major 

Resolved 

Look  at  .org  pw  functions 

New  Feature 

Major 

Resolved 

At  Ghost  Map  Classic  address  (ghostmap.verigames.com),  add  landing  page 

Task 

Blocker 

Resolved 

Point  binaryfission.verigames.com  to  54.80.74.119 

Environment 

Setup 

Blocker 

Resolved 

[Binary  Fission]  Turn  off  web  proxy/pipeline  behavior  for 
binaryfission.verigames.COM 

Bug 

Blocker 

Resolved 

[Binary  Fission]  Cannot  get  Disqus  token  from  verigames.com  endpoint. 

Task 

Major 

Open 

New  Games  page  redesign 

Task 

Major 

Open 

Implement  Achievements/Leaderboards  page 

Task 

Major 

Open 

Add  Attached  Videos  to  Citizen  Science  Page 

Task 

Major 

Open 

Add  Attached  Bibliography  to  Citizen  Science  Page 

Task 

Major 

Open 

Add  Attached  Media  Articles  to  Citizen  Science  Page 

Task 

Major 

Open 

Add  Attached  Blog  Posts  to  Citizen  Science  Page 

Bug 

Major 

Resolved 

api.monsterproof.verigames.com  gives  an  error  -  connection  refused. 

New  Feature 

Major 

Resolved 

Phase  2  Assets:  Monster  proof 

Environment 

Setup 

Critical 

Open 

Apply  security  updates  on  m2.verigames.org 

Task 

Major 

Resolved 

Create  F&F  URLs  and  exclude  from  indxexing 

Bug 

Major 

Resolved 

Setup  Dynamo  Production  Servers 

New  Feature 

Major 

Resolved 

Link  from  hyperspace.ghostmap.com  to 
http://ghostmap.verigames.com/pbgserver/  hyperspace.html 

Bug 

Blocker 

Resolved 

Set  up  Monster  Proof  Production  Game  Server 

Bug 

Major 

Resolved 

502  error  when  posting  to  Gaming  API 

Bug 

Blocker 

Resolved 

Set  up  Monster  Proof  Production  Verification  Servers 

Bug 

Major 

Resolved 

Add  "a.dynamo.verigames.com"  to  the  Domain  Name  System  (DNS)  and 
route  file  to  forward  to  "10.1.4.85:3000" 

Task 

Blocker 

Resolved 

OAuth  not  working  on  Monster  Proof  Production 

Bug 

Blocker 

Resolved 

Cannot  finish  authenticating 

Bug 

Major 

Resolved 

Dynamo  Production  'backend'  server  -  private  key  refused 
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Issue  Type 

Priority 

Status 

Summary 

Task 

Major 

Resolved 

UW  requirements  for  Friends  and  Family 

Task 

Major 

Resolved 

Setup  -  external  names  of  verification  Hosts 

Task 

Major 

Resolved 

Fwd:  Binary  Fission  Disqus 

Task 

Major 

Resolved 

Re:  api.stormbound.verigames.com  is  down 

Task 

Major 

Resolved 

Re:  api.stormbound.verigames.com  is  down 

Task 

Major 

Resolved 

Re:  Logging  in  on  Staging 

Sub-task 

Major 

Resolved 

CSFVDEV-431 

Authentication  for  Phase  II  verification  servers 

Task 

Major 

Resolved 

Get  Lenny  and  Sean  Details  to  the  Data  collection  API  requested  by  Will 

Bug 

Major 

Resolved 

Get  Disqus  token  from  the  server 

Task 

Blocker 

Resolved 

api.stormbound.verigames.com  is  down 

Task 

Major 

Resolved 

Re:  Logging  in 

Task 

Major 

Resolved 

Re:  Logging  in 

Task 

Major 

Resolved 

0Auth2  -  what  is  our  “client  secret”  number? 

Task 

Major 

Resolved 

Verigames  API  bug 

Task 

Major 

Resolved 

Re:  Verigames  API  bug 

Task 

Major 

Resolved 

Using  iPad  app 

Bug 

Major 

Resolved 

Verigames.org  password  reset  doesn't  work-  clicking  email  link  displays  an 
error 

Bug 

Major 

Resolved 

Dynamo  -  OAuth  post  returns  SSL  Certificate  problem 

Task 

Major 

Resolved 

Phase  2  testing  environment  for  Paradox 

Task 

Major 

Resolved 

Correct  two  grammatical  errors  in  verigames.com  FAQ  section 

Environment 

Setup 

Critical 

Resolved 

Apply  security  updates  on  m2.verigames.org 

New  Feature 

Major 

Resolved 

Phase  2  Assets:  Monster  proof 

Bug 

Major 

Resolved 

monsterproof  gaming  api  gives  different  responses  than  the  stormbound  api 

Bug 

Major 

Resolved 

Dynamakr  -  Scores  search  ignores  input  parameters 

Bug 

Blocker 

Resolved 

This  ticket  (and  all  others)  is  publicly  visible 

Task 

Major 

Resolved 

Implement  Achievements/Leaderboards  page 

Task 

Major 

Resolved 

New  Games  page  redesign 

Bug 

Major 

Resolved 

Problem  adding  score  records  to  api.dynamo.verigames.org  -  error  400 

Bug 

Major 

Resolved 

502  bad  gateway  for  api. binaryfission.verigames.com/api/me 

Helpdesk 

Ticket 

Major 

Resolved 

After  /verifySession,  what  should  we  do  with  userid? 

Task 

Major 

Resolved 

Update  verigames.com  for  Phase  2:  Update  Copyright  Info  on 
verigames.com  Footer 

Task 

Major 

Resolved 

Update  verigames.com  for  Phase  2:  Add  New  Games  Categories  to  Forums 

Task 

Major 

Resolved 

Update  verigames.com  for  Phase  2:  Delete  Awards  Module  on  Flow  Jam 
Games  Page 

Bug 

Minor 

Resolved 

"Cannot  upload  photo”  error  on  verigames.com 

Task 

Major 

Resolved 

Create  Forum  categories  for  Beta  testing 

Bug 

Major 

Resolved 

Dynamo  -  Can't  get  user  info  using  "Resource  Owner  Credentials  Grant" 

Oauth.  Returns  "not  authorized" 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Major 

Resolved 

The  login  page  as  part  of  Oauth  redirect,  does  NOT  include  a  'forgot 
password'  link.  Big  hurdle  for  friends  and  family 

Bug 

Major 

Resolved 

Error  returned  when  calling  Scoring  api  -  400  The  query  parameter 
"playerlds"  should  be  valid  array  of  strings... 

Bug 

Major 

Resolved 

Look  at  .org  pw  functions 

New  Feature 

Major 

Resolved 

At  Ghost  Map  Classic  address  (ghostmap.verigames.com)  add  landing  page 

Task 

Blocker 

Resolved 

Point  binaryfission.verigames.com  to  54.80.74.119 

Environment 

Setup 

Blocker 

Resolved 

[Binary  Fission]  Turn  off  web  proxy/pipeline  behavior  for 
binaryfission.verigames.COM 

Bug 

Blocker 

Resolved 

[Binary  Fission]  Cannot  get  Disqus  token  from  verigames.com  endpoint. 

Bug 

Major 

Resolved 

api.monsterproof.verigames.com  gives  a  error  connection  refused. 

New  Feature 

Major 

Resolved 

Link  from  hyperspace.ghostmap.com  to 
http://ghostmap.verigames.com/pbgserver/hyperspace.html 

Task 

Blocker 

Resolved 

OAuth  not  working  on  Monster  Proof  Production 

Bug 

Blocker 

Resolved 

Cannot  finish  authenticating 

Bug 

Major 

Resolved 

test 

Task 

Major 

Resolved 

Fwd:  Binary  Fission  Disqus 

Task 

Major 

Resolved 

Re:  **  PROBLEM  alert  -  stormbound.verigames.com/HTTP  ** 

Task 

Major 

Resolved 

Re:  Bug  in  your  bug-finding-game's  website 

Task 

Major 

Resolved 

Re:  Bug  in  your  bug-finding-game's  website 

Task 

Major 

Resolved 

Re:  Bug  in  your  bug-finding-game's  website 

Bug 

Major 

Resolved 

MTurk  API  returning  "502  Bad  Gateway"  when  calling  /generateToken 

Task 

Major 

Resolved 

Mturk  verigames  access 

Bug 

Major 

Resolved 

Requesting  OAuth  access  token  for  MTurk  API  for  Dynamakr 

Task 

Major 

Resolved 

Requesting  access  to  https://mturk.verigames.org 

Task 

Major 

Resolved 

Fwd:  **  PROBLEM  alert  -  stormbound.verigames.com/HTTP  ** 

Task 

Major 

Resolved 

Register  bug  in  IE11 

Bug 

Minor 

Resolved 

Notification  links  take  you  to  wrong  URL 

Task 

Major 

Resolved 

Replace  current  verigames.com  "About  Us"  page  text  with  attached  text 

Task 

Major 

Resolved 

Remove  "Press  Kit"  From  Links  in  verigames.com  Header  and  Footer 
Navigation 

Bug 

Trivial 

Resolved 

User  icon  doesn't  show  up  in  Authorize  window 

Bug 

Blocker 

Resolved 

Restart  TO  servers  for  StormBound 

Task 

Major 

Resolved 

Request  access  to  mturk  web  dashboard 

Task 

Major 

Resolved 

Change  the  Word  "Interview"  to  :Videos"  in  the  Citizen  Science  pages 

Task 

Major 

Resolved 

Delete  Podcast/Audio  Module  on  Citizen  Science  Pages  and  Replace  with 
Additional  Video  Content 

Task 

Major 

Resolved 

Access  verigames.com  Citizen  Science  Pages  from  Main  verigames.com 
Website 

Improvement 

Major 

Resolved 

Excessive  visits  to  Authorization  page  when  starting  Binary  Fission 

Bug 

Blocker 

Resolved 

api.monsterproof.verigames.com  gives  an  error  when  requesting  score 

Task 

Major 

Resolved 

Add  "Play  Now"  Buttons  Throughout  verigames.com  Citizen  Science  Pages 

Task 

Major 

Resolved 

Make  the  word  "Articles"  Plural  in  the  "Scholarly  Articles"  Heading  in  the 

Citizen  Science  Pages 
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Issue  Type 

Priority 

Status 

Summary 

Task 

Major 

Resolved 

Please  switch  infographics  on  verigames.com  Citizen  Science  pages 

Bug 

Major 

Resolved 

monsterproof.verigames.org  rejecting  root  login 

Task 

Major 

Resolved 

Create  "Play  Now"  Button  for  verigames.com  Citizen  Science  Pages 

Task 

Major 

Resolved 

Delete  Greek  Text  on  Citizen  Science  Page 

Task 

Major 

Resolved 

Images  for  Citizen  Science  Blog  Post  Section 

Bug 

Critical 

Resolved 

1  Agree  Button  Not  Activating  Correctly  for  New  Account 

Bug 

Blocker 

Resolved 

monsterproof  staging  oauth  not  returning  auth  code 

Bug 

Major 

Resolved 

502  bad  gateway  for  oauth.verigames.org/oauth2/authorize 

Task 

Critical 

Resolved 

Create  additional  VMs  for  "monster  proof 

Task 

Major 

Resolved 

Additional  staging  server  HTTPS  proxy  for  MonsterProof 

Bug 

Major 

Resolved 

Mturk  api:  create  hit  complaining  that  "url  should  be  a  string"  in  400  bad 
request 

Task 

Critical 

Resolved 

Deployverigames.com  Advertising  Banner  ASAP  for  Binary  Fission 
Wednesday  July  1st  Event 

Task 

Major 

Resolved 

Add  Monster  Proof  Approximate  Lines  of  Code  Verified  Feed  to  Citizen 

Science  Home  Page 

Task 

Major 

Resolved 

Need  cross  domain  file  to  allow  calls  from  game  page  swf  to  turk  api: 
https://mturk-api.verigames.org/crossdomain.xml 

Bug 

Major 

Resolved 

PHP  scripts  not  running  on  Paradox  staging  server 

Task 

Major 

Resolved 

Mechanical  Turk  -  Add  interface  to  Hit  Create  Page  to  change 
AutoApprovalDelaylnSeconds 

Bug 

Major 

Resolved 

Mturk  when  creating  hit  URLs  the  ?token=####  is  being  added  even  if 
querystring  params  already  exist  in  the  specified  URL 

Bug 

Major 

Resolved 

Dynamakr  Mechanical  Turk  -  Turk  submission  shows  empty  Task  Id 

Task 

Major 

Resolved 

Need  http://ec2-184-73-33-59.compute-l.amazonaws.com/turk/  to  be  visible 

Bug 

Major 

Resolved 

vs4.monsterproof.verigames.com  pointing  to  incorrect  server 

Bug 

Major 

Resolved 

Mturk  Api:  Attempt  to  DELETE  hit  by  hit  id  yielded  Internal  Server  Error  and 
"Hit  with  id=XXX  does  not  exist"  message 

Task 

Major 

Resolved 

Re:  (CSFV  Turk)  502  Bad  gateway  on  turk  API 

Bug 

Major 

Resolved 

Blacklist 

Bug 

Major 

Resolved 

MonsterProof  staging  proxy  HTTPS  certificate  issue 

Task 

Major 

Resolved 

Monster  Proof  Production  is  down 

Task 

Major 

Resolved 

Re:  Access  to  Logaholics? 

Task 

Major 

Resolved 

Re:  Monster  Proof  Production  is  down 

Task 

Major 

Resolved 

Re:  Monster  Proof  Production  is  down 

Task 

Major 

Resolved 

Fwd:  Verigames.com  issue  and  Jira  issue 

Bug 

Blocker 

Resolved 

Monster  Proof  is  getting  Failed  to  load  resource: 
net::ERR  INSECURE  RESPONSE  errors 

Bug 

Minor 

Resolved 

Monster  Proof  Staging  needs  same  routing  setup  as  production 

Task 

Major 

Resolved 

Create  Drop-Down  Banner  for  High  Scorer  Contest 

Task 

Major 

Resolved 

OAuth  servers  down  for  Monsterproof? 

Bug 

Major 

Resolved 

Add  Verigames  Logo  to  High  Score  Contest  News  Post 

Bug 

Major 

Resolved 

Add  High  Scorer  Contest  Rules  to  verigames.com 

Task 

Major 

Resolved 

Please  fix  the  following  errors  on  verigames.com  Citizen  Science  page 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Major 

Resolved 

Error  popup  -  Achievements  page 

Task 

Major 

Resolved 

Technical  problems  Chris 

Task 

Major 

Resolved 

Account  management  samuel.yeom 

Task 

Major 

Resolved 

Remove  verigames.com  drop-down  banner  for  now 

Task 

Major 

Resolved 

Dynamakr  -  Add  a  "Dynamakr  Community  Stats"  panel  to  the  Citizen  Science 
Page 

Task 

Major 

Resolved 

Update  verigames.com  Drop-Down  Banner 

Bug 

Major 

Resolved 

MTurk  qualification  requirement  not  being  properly  added  upon  HIT  creation 

Bug 

Major 

Resolved 

Mechanical  Turk  front  end  panel-  View  Assignments  -  Clicking  column 
headers  does  NOT  re-sort  records 

Bug 

Major 

Resolved 

Mechanical  Turk  -  View  Events  -  Date  column  incorrect  (shows  51  days  ago) 

Bug 

Major 

Resolved 

Mechanical  Turk  -  View  Assignments  -  Does  NOT  display  more  than  10  HITS 

Bug 

Major 

Resolved 

Dynamakr  -the  'How  to  Play'  info  graphic  is  positioned  to  high,  vertically.  The 
top  of  the  words  'How  To  play  the  game'  are  cut  off 

Bug 

Major 

Resolved 

Mechanical  Turk  -  Error  during  Create  Hit  -  insufficient  funds 

Bug 

Major 

Resolved 

Dynamakr  Mechanical  Turk  -  Need  to  change  HIT  description 

Task 

Major 

Resolved 

Other  Paul  Rubens 

Task 

Major 

Resolved 

Other  Paul  Rubens 

Bug 

Major 

Resolved 

Mturk  qualifications  not  being  granted  with  autoApprove 

Bug 

Major 

Resolved 

Turk  API  getAssignment  /  grant  bonus  returning  500  error 

Bug 

Blocker 

Resolved 

Monster  Proof  videos  don't  fit  on  page 

Bug 

Major 

Resolved 

Mturk  api:  create  hit  complaining  that  "url  should  be  a  string"  in  400  bad 
request 

Bug 

Major 

Resolved 

[Forumjuser  can  upload  any  file 

Bug 

Major 

Resolved 

[Forum][286]SQL  injection  Work 

Bug 

Major 

Resolved 

[Forum] [272]SQL  injection  Work 

Bug 

Trivial 

Resolved 

[Profile]  Layout  Issue  when  we  edit  email  notifications 

Bug 

Major 

Resolved 

[Forum]Email  validation  not  done 

Bug 

Trivial 

Resolved 

[Profile]  'Messages'  Tab  is  not  working 

Bug 

Major 

Resolved 

[Forum]Search  Not  Working  Properly 

Bug 

Major 

Resolved 

[Forum][318]  visitor  can  download  the  attachment 

Bug 

Trivial 

Resolved 

[Home]  Games  Menu  overlap  with  the  Social  Media  content  set 

Bug 

Major 

Resolved 

[news][134][135][140][141]Like  button  missing 

Bug 

Major 

Resolved 

[Forum][293]Show  result 

Bug 

Major 

Resolved 

[Forum][247]Show  result 

Bug 

Major 

Resolved 

[Profile]  You  can  add  yourself  as  a  friend 

Bug 

Major 

Resolved 

[Login]  Remember  me  not  working 

Bug 

Trivial 

Resolved 

[Dynamakr]  Clicking  on  SIGN  IN  won't  load  it  as  a  pop  up  (as  in  other  tabs) 

Bug 

Minor 

Resolved 

[Home]  No  Space  in  between  Twitter  Name  and  Twitter  button 

Bug 

Major 

Resolved 

[dynamakr]  Screenshot  status  Overlap  with  Back  button 

Bug 

Trivial 

Resolved 

[News/Games]  Same  section  but  the  functionality  is  different 

Bug 

Trivial 

Resolved 

[Home]  Home  page  slider  is  not  smooth  in  FF 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Major 

Resolved 

[WHAT'S  HOT?]Comment  icon  missing 

Bug 

Major 

Resolved 

[Home]  [Searchjsearch  list  is  not  closing 

Bug 

Major 

Resolved 

[Games][39]  "Search  Members"  text  field 

Bug 

Trivial 

Resolved 

[Citizen  Science]  URLs  are  not  clickable 

Bug 

Major 

Resolved 

[Citizen  Science]  [AIIJBanner  is  too  short  With  Play  button 

Bug 

Major 

Resolved 

[Citizen  Science]  [interviews]  [blog]  [media]  twitter  Scroll  bar  is  disabled 

Bug 

Minor 

Resolved 

[Forum]  Advanced  Search  is  not  working 

Bug 

Major 

Resolved 

Forun  Recent  Post  should  be  in  ascending  order 

Bug 

Major 

Resolved 

Signout  is  enabled  even  though  the  user  plays  as  a  guest 

Bug 

Major 

Resolved 

[Citizen  Science][AII]  Text  overlaps  videos 

Bug 

Major 

Resolved 

[Citizen  Science][Blog]  None  of  the  links  work 

Bug 

Major 

Resolved 

[Footer][183]  Verigames  hyperlink  is  incorrect 

Bug 

Trivial 

Resolved 

[Citizen  Science][AII]  'Verification  facts  discovered'  section  overlap  with  each 
other 

Bug 

Trivial 

Resolved 

[Citizen  Science][Blog]  'View  More'  button  is  not  working 

Bug 

Major 

Resolved 

INFOGRAPHICS  moves  out  of  screen 

Bug 

Major 

Resolved 

Screenshot  number  overlaps  the  back  button  image 

Bug 

Major 

Resolved 

Videos  do  not  play 

Bug 

Trivial 

Resolved 

[Forum]0  Layout  Issue  in  the  WATCHED  button 

Bug 

Major 

Resolved 

[Sign-up  page]  it  should  ask  user  to  enter  the  Password  once  again  while 
signing  up 

Bug 

Major 

Resolved 

[Forum]0  Can't  DELETE  a  single  post  reply  in  a  thread 

Bug 

Major 

Resolved 

[Forum][lll-112]  Can't  DELETE  a  thread 

Bug 

Major 

Resolved 

[Send  Message]  chat  is  not  working 

Bug 

Trivial 

Resolved 

[Forum][129]  Can't  PIN  a  thread 

Bug 

Major 

Resolved 

[Forum][177]  Can't  lock  a  thread 

Bug 

Major 

Resolved 

[Password  recovery]  cannot  receive  password  recovery  email 

Bug 

Major 

Resolved 

[Password  recovery  success  page]  a  dot  should  be  removed  in  the 
confirmation  message 

Bug 

Major 

Resolved 

[Contact-us  page]  the  text  box  can  be  dragged  out  from  the  form  boundary 

Bug 

Major 

Resolved 

[Home  page]  contest  rules  page  cannot  be  accessed 

Bug 

Major 

Resolved 

[News  page]  long  values  in  the  dropdown  list  cannot  be  displayed  fully 

Bug 

Minor 

Resolved 

Confirmation  message  not  valid  for  email  subscribe 

Bug 

Major 

Resolved 

After  Login  won't  redirect  the  page  to  Main  Page 

Client  Task 

Critical 

Resolved 

Need  account  information  for  contest  winners. 

Bug 

Major 

Resolved 

CSFV  Event  BAnner  Design  Task 

Bug 

Major 

Resolved 

fix  chat  configuration 

Bug 

Major 

Resolved 

CSFV  Merge  Branches 

Bug 

Major 

Resolved 

[sdgun]  IE  Bug  Hunts  -  Additional  Payment 

Bug 

Major 

Resolved 

[Saxophonist)]  IE  Bug  Hunts  -  Additional  Payment 

Bug 

Major 

Resolved 

[macs054]  IE  Bug  Hunts  -  Additional  Payment 

Bug 

Major 

Resolved 

[dpebble]  IE  Bug  Hunts  -  Additional  Payment 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Major 

Resolved 

[karthikbecse]  IE  Bug  Hunts  -  Additional  Payment 

Bug 

Major 

Resolved 

Fixed  Styling  in  Science  page 

Bug 

Major 

Resolved 

[Forumjuser  can  upload  any  file 

Bug 

Major 

Resolved 

[Forum][286]SQL  injection  Work 

Bug 

Major 

Resolved 

Social  Media  Promotion  Payments :  July  2015 

Bug 

Major 

Resolved 

Social  Media  Promotion  Payments :  August  2015 

Bug 

Major 

Resolved 

After  loading  High  score  configuration,  the  score  is  different  in  the  game 
section 

Bug 

Major 

Resolved 

When  the  map  zone  is  over  buttons  these  cannot  be  used 

Bug 

Major 

Resolved 

While  loading  the  game  the  scores  are  shown  as  more  than  100%  some 
times  which  is  incorrect 

Bug 

Trivial 

Resolved 

[CommonjjLevels]  When  we  Zoom  In;  Optimizer  selection  circles  won't  zoom 

In 

Bug 

Major 

Resolved 

Annoying  text  area  -  High  scores  Click  on  a  score  to  load  it  -  Appears  every 
few  minutes 

Bug 

Major 

Resolved 

Lost  zoom  out  function 

Bug 

Major 

Resolved 

Game  not  Loading  on  Firefox 

Bug 

Minor 

Resolved 

A  value  of  score  improvements  is  negative 

Bug 

Minor 

Resolved 

Scores  are  more  than  100% 

Bug 

Minor 

Resolved 

Score  with  title 

Bug 

Minor 

Resolved 

Duplicate  Username  in  Achievement  High  Scores 

Bug 

Major 

Resolved 

There  are  some  levels  in  which  even  after  we  reach  100%,  it  doesnt  show  the 
level  completed  message 

Bug 

Major 

Resolved 

odd  intermediate  values  displayed  during  level  loading 

Bug 

Major 

Resolved 

unable  to  restart  a  level  from  the  beginning 

Bug 

Trivial 

Resolved 

[Common][Levels]  Marker  must  display  in  front  of  the  Level  Number 

Bug 

Major 

Resolved 

Tutorial  button  sometimes  not  shown  on  home  page 

Bug 

Trivial 

Resolved 

volume  icon  is  too  small  on  home  page 

Bug 

Major 

Resolved 

no  need  to  show  High  Scores  and  Score  Improvements  sections  on  home 
page 

Bug 

Trivial 

Resolved 

spelling  and  text  size  should  be  changed 

Bug 

Major 

Resolved 

Achievement  text  is  not  seen  well 

Bug 

Major 

Resolved 

see  negative  number  for  the  "Constraints  Solved" 

Bug 

Trivial 

Resolved 

[Common]  Graphic  overlap  with  the  Tool  Tip 

Bug 

Trivial 

Resolved 

[Common]  Tool  Tip  still  remains  even  the  Tool  bar  is  gone 

Bug 

Major 

Resolved 

cannot  exit  at  the  beginning  of  the  tutorial 

Bug 

Major 

Resolved 

cannot  finish  tutorial 

Bug 

Minor 

Resolved 

Closing  help  dialog  with  [x]  button  does  not  work 

Bug 

Minor 

Resolved 

Maximum  call  stack  size  exceeded 

Bug 

Trivial 

Resolved 

[Common]  Info  boxes  must  center  align  with  the  screen  area 

Bug 

Trivial 

Resolved 

[Common]  'Enhance  Energy'  Tool  tip  displayed  most  of  the  time 

Bug 

Major 

Resolved 

[Common]  Very  difficult  to  play  the  game  in  Chrome 

Bug 

Major 

Resolved 

patterns  appears  hidden 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Trivial 

Resolved 

[Common]  Top  Menu's  partly  hidden  even  in  Full  screen  in  Chrome 

Bug 

Major 

Resolved 

Hint  doesn't  disappear 

Bug 

Trivial 

Resolved 

[Common]  Error  while  playing  the  game 

Bug 

Trivial 

Resolved 

[Common]  Error  displayed  while  clicking  Menu  when  playing 

Bug 

Major 

Resolved 

Tools  Up/Down  button  does  not  have  any  effect 

Bug 

Trivial 

Resolved 

[Common]  Award  Images  not  displayed 

Bug 

Minor 

Resolved 

Close  button  doesn't  work 

Bug 

Trivial 

Resolved 

[Common]  'Anount'  must  -->  Amount 

Bug 

Trivial 

Resolved 

[Common][Levels]  When  User  set  a  high  Score  [If  there  is  no  Other  High 
Scores]  It  won't  update  the  'High  Scores'  section 

Bug 

Trivial 

Resolved 

[Common][Levels]  Over  100%  values  displayed  when  loading 

Bug 

Minor 

Resolved 

[Common][Levels]  High  Scores  updated  but  it  is  not  updated  in  the  Game 
section 

Bug 

Major 

Resolved 

Volume  button  should  not  be  seen  in  menu  screen.  Sometimes  it  appears 

Bug 

Trivial 

Resolved 

[Common][Levels]  Layout  Issue  in  the  Play/Pause  Music  Button 

Bug 

Major 

Resolved 

the  screen  has  grids  while  zoom  in/out 

Bug 

Major 

Resolved 

Game  stuck 

Bug 

Trivial 

Resolved 

[Common]  Error  occurs  while  playing  the  tutorials 

Bug 

Trivial 

Resolved 

Top  Part  (Username,  Energy,  Pattern  and  Size)  is  not  clearly  visible  in  normal 
mode 

Bug 

Minor 

Resolved 

paintbrush  should  not  be  shown  if  we  click  the  "Next  Level"  button 

Bug 

Major 

Resolved 

[Common][Levels]  If  we  don't  press  NEXT  from  pop  up,  can't  go  to  next  level 

Bug 

Minor 

Resolved 

[Tutorials]  Mini  Map  location  instruction  Issue 

Bug 

Minor 

Resolved 

[Tutorials]  Text  overlap  with  the  Buttons 

Bug 

Minor 

Resolved 

[Tutorials]  Zoom  In/Out  are  not  working 

Bug 

Minor 

Resolved 

"Ghost  filter"  is  applied 

Bug 

Trivial 

Resolved 

Empty  chat  msgs  also  gets  posted 

Bug 

Minor 

Resolved 

Game  is  fully  accessed  by  web  toolkit 

Bug 

Major 

Resolved 

Particles  disappears  and  score  goes  to  0 

Bug 

Major 

Resolved 

Clicking  the  completed  mission  shows  information  for  another  mission 

Bug 

Major 

Resolved 

Game  was  frozen 

Bug 

Major 

Resolved 

Some  functionalities  are  not  explained  anywhere  -  e.g.  Utility  Balance  and 

Purity 

Bug 

Major 

Resolved 

Redirection  after  1  pasted  http://binaryfission.verigames.com/ 

Bug 

Minor 

Resolved 

Color  of  some  cells  are  strange 

Bug 

Trivial 

Resolved 

[Common]  Score  Card  displays  when  we  go  to  Game  page 

Bug 

Major 

Resolved 

Limit  chat  input 

Bug 

Trivial 

Resolved 

[Common]  Text  Issue  in  FF 

Bug 

Trivial 

Resolved 

[Common]  Layout  issue  in  Top  Scores  section  in  FF 

Bug 

Trivial 

Resolved 

[Common]  Layout  issue  in  Borders  of  Puzzles  in  FF 

Bug 

Major 

Resolved 

playing  a  specific  level  in  Puzzles  section  is  loading  slowly 

Bug 

Trivial 

Resolved 

volume  control  button  is  not  necessary  in  the  Lobby  page 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Trivial 

Resolved 

content  should  be  changed  in  the  Puzzles  section 

Bug 

Minor 

Resolved 

better  add  a  X  button  for  the  modal 

Bug 

Major 

Resolved 

Chat  area  should  show  timestamp 

Bug 

Major 

Resolved 

user  info(level  and  points)  is  not  updated  instantly  on  the  Lobby  page 

Bug 

Trivial 

Resolved 

[Common]  'Record  Holder'  section  Layout  Issue  in  IE11 

Bug 

Trivial 

Resolved 

Ivl  should  be  changed 

Bug 

Trivial 

Resolved 

last  two  lines  should  have  more  vertical  distance 

Bug 

Trivial 

Resolved 

spelling  in  the  basic  info  area  should  be  adjusted 

Bug 

Minor 

Resolved 

CHat  Window  Layout  issue  in  IE11 

Bug 

Major 

Resolved 

the  value  of  "Number  of  Solutions  Found”  is  not  increased  after  replaying  the 
level 

Bug 

Major 

Resolved 

capitalization  should  be  consistent  on  the  scorecard 

Bug 

Minor 

Resolved 

should  show  basic  user  info  on  the  level  playing  page 

Bug 

Major 

Resolved 

"Skip"  is  working  slowly 

Bug 

Minor 

Resolved 

There  is  no  indication  to  the  user  once  the  puzzle  is  solved 

Bug 

Trivial 

Resolved 

Level  20  description  overlaps  next  line 

Bug 

Major 

Resolved 

Multiple  Levels  Loading  Simultaneously 

Bug 

Major 

Resolved 

One  Circle  is  overlapping  with  the  other  one  at  the  end 

Bug 

Major 

Resolved 

f(x)-balance  doesn't  show  permanently  sometimes 

Bug 

Trivial 

Resolved 

extra  dot  in  the  sentence  should  be  removed 

Bug 

Major 

Resolved 

should  not  allow  send  empty  message 

Bug 

Trivial 

Resolved 

Can  Insert  Empty  messages 

Bug 

Major 

Resolved 

long  value  cannot  be  show  fully  (should  restrict  the  max  limit  of  the  text) 

Bug 

Trivial 

Resolved 

Chat  message  display  delay 

Bug 

Trivial 

Resolved 

Horizontal  Scroll  bars  appears  in  the  Puzzles  area 

Bug 

Trivial 

Resolved 

'Score'  goes  beyond  the  container  area 

Bug 

Minor 

Resolved 

Close  button  layout  issue 

Bug 

Minor 

Resolved 

'Score'  Overlap  with  the  bottom  border 

Bug 

Trivial 

Resolved 

[Cassiopeia][Delta  Cassiopeiae]  Error  Occurred  while  playing  the  game 

Bug 

Minor 

Resolved 

[Aquila][Gamma  Aquilae]  Can't  complete  the  Level 

Bug 

Minor 

Resolved 

[Aquila][Alpha  Aquilae]  Can't  complete  the  Level 

Bug 

Trivial 

Resolved 

[Pavo][Alpha  Pavonis]  Can't  complete  the  Level 

Bug 

Major 

Resolved 

Alpha  Phoenicis  -  Game  crashed  when  playing  this  level 

Bug 

Minor 

Resolved 

[Cassiopeia][Delta  Cassiopeiae]  Error  Occurred  while  Restarting  the  game 

Bug 

Major 

Resolved 

[Pavo][Beta  Pavonis]  Can't  complete  the  Level 

Bug 

Major 

Resolved 

Levels  with  lot  of  planets  can  never  be  completed  because  of  the  fact  that 
there  are  n  number  of  invaders  appearing  all  at  once 

Bug 

Major 

Resolved 

History  is  not  displayed  if  user  is  playing  the  same  level  for  the  second  time, 
however  it  checks  for  duplicates  and  shows  appropriate  message 

Bug 

Major 

Resolved 

Sometimes  invaders  are  not  appearing  but  only  bursting-animation  is  shown 

Bug 

Major 

Resolved 

Error  while  playing  Alpha  Pavonis 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 

36 


Issue  Type 

Priority 

Status 

Summary 

Bug 

Major 

Resolved 

Beta  Aquilae  -  Not  able  to  complete  this  level.  Tried  all  combinations  of  the 

Rift  nodes 

Bug 

Major 

Resolved 

IDEA  tool  appears  on  the  planets  screen  when  we  exit  out  of  the  game 

Bug 

Major 

Resolved 

Alpha  Circini  after  first  rift  sealed 

Bug 

Major 

Resolved 

unable  to  select  any  section 

Bug 

Major 

Resolved 

tried  sections  not  played 

Bug 

Major 

Resolved 

[Something  that  doesn't  make  sense]  -  Meaning  of  the  "Bar  with  a  moving 
arrow"  in  the  game-play  area  is  not  understandable 

Bug 

Major 

Resolved 

Delta  Circini  -  Bounty  is  not  reset  when  the  level  is  restarted 

Bug 

Major 

Resolved 

Game  disappeared  and  a  blank  white  screen  came  up  when  playing 

Bug 

Major 

Resolved 

Delta  Circini  -  Not  able  to  set  the  Rift  to  a  particular  path.  It  keeps  on  resetting 
to  another  path. 

Bug 

Major 

Resolved 

[Something  that  doesn't  make  sense]  -  After  completing  a  level,  user  has  to 
manually  exit  the  current  level  to  move  on  to  next  level 

Bug 

Major 

Resolved 

Gamma  Cassiopeiae  -  "Cannot  read  property  getTraceOutEdge"  Error 
displayed  after  zapping  an  invader 

Bug 

Major 

Resolved 

Level  is  marked  as  complete  though  player  has  failed  the  level 

Bug 

Trivial 

Resolved 

[Serpens][Alpha  Serpens]  I.D.E.A  Section  is  not  displayed 

Bug 

Major 

Resolved 

Game  level  name  goes  outside  the  planet  boundary 

Bug 

Major 

Resolved 

Game  level  name  overlaps  with  the  Planet  name 

Bug 

Major 

Resolved 

Showing  all  og  the  levels  inside  Centaurus  as  complete 

Bug 

Minor 

Resolved 

Dragging  the  map  to  the  rightmost  screen  causes  the  page  to  expand. 

Bug 

Major 

Resolved 

Delta  Circini  -  Not  able  to  complete  this  level  though  1  have  tried  almost  every 
part  of  the  Rift  (more  than  20  attempts) 

Bug 

Major 

Resolved 

Information  message  displayed  during  the  game-play  does  not  get  cleared 
even  when  the  user  is  in  map 

Bug 

Minor 

Resolved 

[Cassiopeia][Alpha  Cassiopeiae]  Clicking  on  'UNLOCK  ALL'  won't  clear  the 
Crossed  Paths 

Bug 

Major 

Resolved 

Incorrect  Credit  is  shown  when  the  level  is  restarted 

Bug 

Major 

Resolved 

[Cassiopeia][Delta  Cassiopeiae]  Clicking  on  'Auto  Pilot'  will  Exit  the  Level 

Bug 

Major 

Resolved 

Unable  to  play 

Bug 

Minor 

Resolved 

[Common]  Opening  the  game  begins  at  the  tutorial  even  if  the  tutorial  is 
completed 

Bug 

Major 

Resolved 

[Cassiopeia][Beta  Cassiopeiae]  Blank  Black  Area  displayed  at  the  bottom 
when  we  complete  the  Level 

Bug 

Minor 

Resolved 

[Cassiopeia][Beta  Cassiopeiae]  Error  Occurred  while  playing  the  game 

Bug 

Minor 

Resolved 

[Common][Tutorial][Sectors]  Return  to  Main  Menu  Tool  Tip  Issue 

Bug 

Minor 

Resolved 

[Common][Tutorial]  After  completing  a  Tutorial;  Is  won't  display  as  Completed 

Bug 

Trivial 

Resolved 

unnecessary  tooltip  should  be  removed  for  "Click  for  Next  Level" 

Bug 

Trivial 

Resolved 

"Zoom  In  /  Zoom  Out"  should  be  "ZOOM  IN  /  ZOOM  OUT" 

Bug 

Major 

Resolved 

Instead  of  closing  IDEA  moves  down 

Bug 

Major 

Resolved 

Window  position-IDEA  and  game  overlay  doesn't  fit  well  -  screen  resolution 

Bug 

Trivial 

Resolved 

[Common][Tutorial]  Sealing  :  Text  go  beyond  the  container  area 

Bug 

Minor 

Resolved 

[Common]  Content  overlap  when  we  move  the  graph  in  Rift  Node  chart 

Bug 

Minor 

Resolved 

[Common]  Click  on  I.D.E.A  Title  section  will  move  the  pop  up  down 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Trivial 

Resolved 

[Common]  Blank  space  displayed 

Bug 

Major 

Resolved 

[Common]  can't  Play  the  game  in  IE 

Bug 

Trivial 

Resolved 

'Zoom  Out'  Tool  Tip  issue 

Bug 

Minor 

Resolved 

Nothing  happens  when  we  click  on  'Combat'  button 

Bug 

Trivial 

Resolved 

Sometimes  Tutorial  names  overlap  with  the  'Title' 

Bug 

Trivial 

Resolved 

'Unlock  positions  of  all  nodes'  Tool  Tip  won't  clearly  displayed 

Bug 

Trivial 

Resolved 

Exit  to  Main  page  won't  work  properly 

Bug 

Trivial 

Resolved 

Help  Tool  Tip  overlap  each  other 

Bug 

Major 

Resolved 

Design  Contest  Badges 

Copilot 

Payment 

Major 

Resolved 

CSFV-4468 

Copilot  payment 

Bug 

Major 

Resolved 

Badges  Profile  Popup  and  Forums  Fix 

Copilot 

Payment 

Major 

Resolved 

CSFV-4470 

Review  +  Copilot  Payment 

Bug 

Minor 

Resolved 

Things  go  out  of  frame  in  'Else'  part 

Bug 

Trivial 

Resolved 

Tutorial  Player  progress  Bar  is  not  completed 

Bug 

Trivial 

Resolved 

Duplicate  Freebies  detected 

Bug 

Trivial 

Resolved 

All  the  objects  are  not  displayed  in  the  Freebies 

Bug 

Trivial 

Resolved 

[common]  Extra  Credit  button  is  not  working 

Bug 

Major 

Resolved 

When  user  add  'CROSS  ROADS'  it  won't  appear  in  the  game  preview 

Bug 

Major 

Resolved 

Bad  request  -  400  encountered 

Bug 

Major 

Resolved 

Flip  function  is  not  working  properly 

Bug 

Major 

Resolved 

freebies  section  should  not  have  duplicate  items 

Bug 

Trivial 

Resolved 

All  the  objects  are  not  displayed  in  the  Popup  box 

Bug 

Trivial 

Resolved 

Some  items  are  not  fully  displayed 

Bug 

Major 

Resolved 

Sometimes  theres  nothing  in  the  bottom  gate 

Bug 

Major 

Resolved 

Previous  Stash  Icons  displayed  when  we  come  again 

Bug 

Minor 

Resolved 

When  user  add  'CROSS  ROADS'  it  won't  appear  in  the  game  preview 

Bug 

Major 

Resolved 

Even  after  i  unlock  the  monster  hatchery  it  is  not  enabled 

Bug 

Major 

Resolved 

Can't  complete  the  Forest  Loop  Level 

Bug 

Trivial 

Resolved 

Monster  Hatchery  is  not  activated 

Bug 

Major 

Resolved 

Create  unlimited  freebies 

Bug 

Major 

Resolved 

freebies  section  should  be  able  to  be  expanded  larger 

Bug 

Major 

Resolved 

Nothing  happens  on  clicking  on  Take  Class 

Bug 

Major 

Resolved 

Although  logged  in  one  of  the  screen  displays  guest 

Bug 

Major 

Resolved 

Last  two  Levels  (Tutorials)  completed  automatically;  (PR327,  PR333) 

Bug 

Minor 

Resolved 

PR120  video  is  not  playing 

Bug 

Trivial 

Resolved 

[common]  Grade-D  must  display  in  single  line 

Bug 

Major 

Resolved 

PR301:  Required  Course  is  wrong 

Bug 

Major 

Resolved 

PR330:  Required  Course  is  wrong 

Bug 

Major 

Resolved 

Take  Class  is  not  working 
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Issue  Type 

Priority 

Status 

Summary 

Bug 

Major 

Resolved 

the  values  should  be  0  when  there  is  no  Music/Fx 
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Appendix  B.  Expert  Game  Analysis:  Dynamakr 
B.1  Quick  Summary 
B.1.1  Best  Features 

•  Graphics  look  cool  and  the  story/setting  seems  intriguing. 

•  The  shooter  game  is  fun. 

B.1 .2  Needs  Improvement 

•  Need  to  give  the  50,000  ft  view  first:  A  single,  clear  goal  with  steps  to  achieve  it  like  “Enter  the 
dynamo  by  filling  up  one  of  the  green  bars  at  the  top.” 

o  Later  you  can  get  more  nuanced  with  ideas  like  “you  want  to  get  as  much  energy  as 
possible”  and  “linked  nodes  produce  more  energy.” 
o  This  also  applies  to  the  weaver  -  a  single  clear  goal  like  “shoot  everything  in  sight” 
should  be  the  first  thing  you  tell  the  player. 

•  Terminology  does  not  aid  understanding.  Terms  like  pattern,  knot,  and  snarl  don’t  seem  to 
correspond  to  any  of  the  game  graphics. 

•  Lots  of  movement  and  lots  of  clicking  to  achieve  goals.  The  patterns  move  constantly,  which 
makes  it  a  struggle  to  use  the  interface. 

•  There  seems  to  be  a  very  weak  connection  between  what  the  player  does  and  what  happens  with 
the  patterns  on  screen.  It  is  very  difficult  to  predict  what  will  happen  with  any  given  action, 
making  the  game  mechanics  very  difficult  to  learn. 

Table  7  shows  a  description  of  each  area  for  analysis  and  its  priority  for  the  Dynamakr  game. 
Each  of  these  areas  are  addressed  in  the  following  sections. 


Table  7  Areas  of  Analysis  and  Priorities  for  the  Dynamakr  Game 


Area 

Priority 

Usability  questions:  Heuristics 

Visibility  of  system  status 

High 

Match  between  the  system  and  the  real  world 

High 

User  control  and  freedom 

Low 

Consistency  and  standards 

High 

Error  prevention 

Medium 

Recognition  rather  than  recall 

Low 

Flexibility  and  efficiency  of  use 

Low 

Aesthetic  and  minimalist  design 

Low 

Help  users  recognize,  diagnose,  and  recover  from  errors 

Medium 

Help  and  documentation 

Low 

Game  questions 

Tutorial  analysis 

High 

Engagement  analysis 

Medium 

Coherency  analysis 

Medium 

Analysis  of  value  to  player  provided  by  game 

Medium 
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B.2  Usability  Questions:  Heuristics  Analysis 
B.2.1  Visibility  of  System  Status 

The  game  should  keep  players  informed  about  what  is  going  on,  through  appropriate 
feedback  within  reasonable  time.  What  feedback  is  missing? 

•  It  is  unclear  what  the  semi-circular  energy-bar  line  (energy  points?)  around  each  pattern 
represents  and  how  it  is  changed. 

•  It  is  unclear  why  there  is  a  green  number  to  the  right  of  the  green  radiating  hexagons. 

•  It  is  unclear  why  some  patterns  have  white  stars. 

•  It  is  unclear  what  the  various  colors  of  squiggly  lines  indicate. 

B.2.2  Match  between  the  System  and  the  Real  World 

The  game  should  speak  the  player ’s  language,  with  words,  phrases,  and  concepts  familiar  to 
the  player,  rather  than  system-oriented  terms.  Follow  real-world  conventions,  making 
information  appear  in  a  natural  and  logical  order.  What  terms  are  out  of  place  or  poorly  aligned 
with  what  they  represent? 

•  The  color  red  has  a  negative  connotation  that  can  cause  players  to  think  that  there  is  something 
wrong  with  the  patterns,  even  though  a  pattern  might  be  as  optimized  as  possible. 

•  The  graphics  do  not  necessarily  match  the  language,  such  as  snarls,  knots,  etc.  It  is  also  unclear 
how  snarls  and  knots  get  created  or  undone  in  the  game  (other  than  shooting  them). 

•  There  seems  to  be  a  very  weak  connection  between  what  the  player  does  and  what  happens 
with  the  patterns  on  screen.  It  is  very  difficult  to  predict  what  will  happen  with  any  given 
action,  making  the  game  mechanics  very  difficult  to  learn. 

B.2. 3  User  Control  and  Freedom 

Players  often  choose  actions  by  mistake  and  need  a  clearly  marked  “emergency  exit  ”  to 
leave  the  unwanted  state  without  having  to  go  through  an  extended  dialogue.  Support  undo  and 
redo.  What  mistakes  require  too  much  work  to  recover  from? 

•  There  is  no  undo  button  to  go  back  to  a  previous  step.  Even  if  there  is  no  benefit  to  undoing  a 
completed  step  from  a  score  perspective,  there  might  still  be  value  in  undoing  the  automatic 
pattern  rearrangement. 

•  Constant  automatic  shuffling  and  rearranging  the  patterns  is  very  disorienting  and  decreases  a 
player’s  sense  of  control.  It  also  confuses  players  and  makes  the  game  feel  arbitrary. 

B.2.4  Consistency  and  Standards 

Players  should  not  have  to  wonder  whether  different  words,  situations,  or  actions  mean  the 
same  thing.  Follow  platform  conventions.  What  inconsistencies  make  playing  or  understanding 
the  game  harder? 

•  The  game’s  mechanics  are  unclear  to  the  point  that  the  game  can  seem  chaotic,  much  like  a 
kaleidoscope.  Sometimes  it  seems  that  adding  a  new  pattern  one  time  may  cause  the  entire 
screen  to  rearrange  itself  to  the  left,  but  rearrange  itself  to  the  right  on  the  next  addition.  It  is 
very  difficult  to  learn  the  rules  when  it  is  difficult  to  identify  the  consistent  behaviors. 

•  The  game’s  story  refers  to  a  quantum  3D  printer,  but  then  seems  to  ignore  this  central  aspect 
of  the  game. 

B.2. 5  Error  Prevention 

Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a  problem  from 
occurring  in  the  first  place.  Either  eliminate  error-prone  conditions  or  check  for  them  and 
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presen  t  players  with  a  confirmation  option  before  they  commit  to  the  action  .  Does  the  game  let 
me  blunder  into  serious  errors? 

•  May  not  be  applicable  in  the  pattern  portion  of  the  game. 

•  In  the  shooting  part  of  the  game,  it  is  difficult  to  identify  power-ups  or  distinguish  them  from 
enemies.  It  is  too  easy  to  lose  a  powered  up  weapon  when  the  player  accidentally  picks  up  a 
low-level  weapon.  Consider  adding  the  option  to  let  users  reject  an  unwanted  weapon. 

B.2.6  Recognition  instead  of  Recall 

Minimize  the  player ’s  memory  load  by  making  objects,  actions,  and  options  visible.  The 
player  should  not  have  to  remember  information  from  one  part  of  the  game  to  another. 
Instructions  for  use  of  the  system  should  be  visible  or  easily  retrievable  whenever  appropriate. 
Which  instructions  are  difficult  to  find  or  refer  to? 

•  The  graphical  presentation  does  not  require  players  to  recall  vital  infonnation  to  play. 

B.2.7  Flexibility  and  Efficiency  of  Use 

Accelerators — unseen  by  the  novice  player — may  speed  up  the  interaction  for  the  expert 
player  such  that  the  game  can  cater  to  both  inexperienced  and  experienced  players.  Allow 
players  to  tailor  frequent  actions.  What  actions  do  I find  repetitive  or  wish  I  could 
hotkey/macro? 

•  Consider  adding  hotkeys  for  the  various  pattern  tools. 

•  Consider  adding  the  ability  for  players  to  change  weapons  during  the  shooting  portion. 

B.2.8  Aesthetic  and  Minimalist  Design 

Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Every  extra 
unit  of  information  in  a  dialogue  competes  with  the  relevant  units  of  information  and  diminishes 
their  relative  visibility.  What  can  be  removed  to  improve  clarity? 

•  The  rotating  ring  around  each  pattern  draws  more  attention  than  the  semi-circle  around  it.  If 
the  semi-circle  contains  important  data,  it  should  be  more  salient  than  the  rotating  ring. 

•  The  jittering  lines  can  become  very  distracting. 

B.2.9  Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors 

Error  or  failure  messages  should  be  expressed  in  plain  language  (no  codes),  precisely 
indicate  the  problem,  and  constructively  suggest  a  solution.  When  I  fail  or  make  a  mistake,  do  I 
understand  why  and  how  to  do  better? 

•  The  game  mechanics,  including  errors,  are  difficult  to  discover,  which  makes  it  difficult  to 
leam  how  to  improve. 

B.2.10  Help  and  Documentation 

Even  though  it  is  better  if  the  game  can  be  used  without  documentation,  it  may  be  necessary 
to  provide  help  and  documentation.  This  information  should  be  easy  to  search,  list  concrete  steps 
to  be  carried  out,  and  not  be  too  large.  What  things  do  I  wish  I  could  look  up  quickly  while 
playing? 

•  It  is  great  that  there  is  a  help  button  in  the  bottom  right  corner;  unfortunately,  the 
documentation  provides  only  limited  support  to  the  player  trying  to  understand  the 
complexities  of  this  game. 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 

42 


B.3  Game  Questions 
B.3.1  Tutorial  Analysis 

Assessment  of  the  effectiveness  of  the  tutorial;  how  effective  it  is  at  teaching  its  topics,  how 
well  you  feel  you  understand  the  game  at  the  end  of  it,  how  well  it  sets  you  up  for  the  jump  to 
real  gameplay,  how  well  it  draws  you  into  the  game.  Did  the  tutorial  help?  What  kind  of  player 
does  it  support?  Would  you  finish  it? 

•  Tutorial  throws  out  a  lot  of  jargon  that  is  tough  to  understand.  Even  where  the  main  ideas  come 
through,  it  is  not  clear  whether  the  player  should  enter  the  dynamo  as  soon  as  possible  or  wait 
a  bit,  or  how  to  make  that  decision  if  it’s  a  trade-off.  None  of  the  effects  of  actions  are  clear 
(beyond  creating  new  patterns  and  marking  them  with  the  tools  like  the  energizer). 

•  Need  to  start  the  player  off  with  a  simple,  clear  goal  and  then  add  nuance  afterwards. 

•  The  tutorial  doesn’t  explain  the  connection  between  the  two  games  or  any  strategy  tips  or 
information.  The  help  pop-ups  are  useful  and  mention  that  connected  patterns  are  worth  more 
energy,  but  this  seems  like  something  that  should  be  more  explicitly  taught,  since  it’s  key  to 
differentiating  the  actions  available. 

B.3.2  Engagement  Analysis 

Assessment  of  overall  enjoyment,  including:  what  elements  make  the  game  enjoyable,  what 
elements  detract.  How  can  elements  be  changed  to  enhance  enjoyment? 

•  The  main  thing  that  has  potential  for  a  fun  game  loop  is  the  ability  to  seed  a  level  with  goodies 
and  then  go  in  and  collect  them. 

•  This  would  be  more  compelling  and  apparent,  though,  if  there  was  some  count  or  weight 
or  whatever  attributed  to  actions  in  the  pattern-finding  game.  For  example,  a  snarl  danger 
level  might  be  shown  on  the  screen  and  energizing  a  pattern  would  reduce  it  immediately 

•  To  close  this  loop,  there  needs  to  be  something  to  spend  the  collected  loot  on  once  the 
player  exits.  This  could  be  as  simple  as  purchasing  further  customization  options,  like 
seeding  in  weapons  and  shields  by  spending  energy  to  add  them  prior  to  entering  the 
weaver. 

B.3. 3  Coherency  Analysis 

Assessment  of  coherency  and  balance  between  elements,  such  as:  mechanics,  story, 
aesthetics,  and  technology.  What  elements  seem  to  conflict  or  undermine  each  other? 

•  The  lack  of  clear  connection  between  the  games  (from  the  player’s  perspective)  prevents  that 
from  being  a  strength.  Some  rewording  to  help  the  player  fit  the  two  games  together 
conceptually  and  some  more  transparency  of  the  mechanics  connecting  them  would  go  a  long 
way  here. 

•  All  the  motion,  animation,  and  flying/zapping  bits  in  the  pattern-finding  game  seem  to 
undermine  the  ability  to  do  anything  with  it.  That,  plus  the  unclear  connections  to  the  other 
game  make  this  feel  a  bit  like  a  cool  data  visualization  instead  of  a  game. 

B.3.4  Analysis  of  Value  to  Player  Provided  by  Game 

What  is  the  player  getting  out  of playing  the  game?  Why  would  they  press  the  “ next  level’’ 
button?  Is  it  fun?  Are  they  making  progress?  Are  they  leveling  up  or  filling  a  bar  or 
something?  Are  their  game-playing  peers  depending  on  them? 

•  There  is  little  progression  of  the  experience — the  bars  get  larger,  but  it  would  be  more 
compelling  if  this  was  tied  to  discrete  “level-up”  events  that  the  player  could  see  coming,  such 
as  “next  level  at:  10,000  XP” 
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•  It  would  also  help  if  more  tools  and/or  powerups  were  unlocked  or  if  there  was  a  sort  of 
“energy  multiplier”  or  some  other  signifier  of  progress 

There  is  no  tie  to  any  extrinsic  motivation,  for  example: 

•  Indications  of  overall  progress  being  made  (this  could  be  the  energy  multiplier,  if  it’s  true 
that  more  passes  by  anyone  tend  to  increase  the  energy  of  moves) 

•  Leaderboards  or  other  forms  of  comparison  to  other  players 

•  Benchmarks  of  any  sort  to  let  the  player  know  whether  or  not  they  did  well  at  a  level.  “You 
got  3750!”  is  less  compelling  than  “You  got  3750  out  of  a  possible  5000!” 
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Appendix  C.  Expert  Game  Analysis:  Binary  Fission 
C.1  Quick  Summary 
C.1.1  Best  Features 

•  Very  playable,  satisfying  game  of  splitting  things  up 

•  The  graphics  are  appealing 

C.1 .2  Needs  Improvement 

•  There  is  little  connection  at  the  moment  to  any  motivational  loop  -  it  is  fun  to  do  for  a  bit,  but 
what  brings  me  back? 

•  Unfinished  pieces  like  algorithms,  power  rings,  end  of  tutorial 

Table  8  shows  a  description  of  each  area  for  analysis  and  its  priority  for  the  Binary  Fission 
game.  Each  of  these  areas  are  addressed  in  the  following  sections. 


Table  8  Areas  of  Analysis  and  Priorities  for  the  Binary  Fission  Game 


Area 

Priority 

Usability  questions:  Heuristics  analysis 

Visibility  of  system  status 

High 

Match  between  the  system  and  the  real  world 

Medium 

User  control  and  freedom 

Low 

Consistency  and  standards 

High 

Error  prevention 

Low 

Recognition  rather  than  recall 

Low 

Flexibility  and  efficiency  of  use 

High 

Aesthetic  and  minimalist  design 

Low 

Help  users  recognize,  diagnose,  and  recover  from  errors 

Low 

Help  and  documentation 

Medium 

Game  questions 

Tutorial  analysis 

Medium 

Engagement  analysis 

Medium 

Coherency  analysis 

Medium 

Analysis  of  value  to  player  provided  by  game 

High 

C.2  Usability  Questions:  Heuristics  Analysis 
C.2.1  Visibility  of  System  Status 

The  game  should  keep  players  informed  about  what  is  going  on,  through  appropriate 
feedback  within  reasonable  time.  What  feedback  is  missing? 

•  There  is  no  connection  between  a  ring  option  and  the  effect  in  the  nucleus,  other  than  that 

certain  colors  are  more  effective  at  separating  quarks. 

•  There  is  a  one-to-many  mapping,  where  the  same  filter  option  icon  could  cause  the  quarks 
to  separate  in  very  different  ways.  This  makes  it  impossible  to  leam  or  predict  the  behavior 
of  each  filter  option  type,  requiring  the  player  to  review  each  of  the  options. 

•  Every  time  a  player  clicks  on  and  off  of  a  nucleus,  they  are  presented  with  a  new  set  of 
options. 
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•  For  players  to  learn  the  game  mechanics  and  behaviors,  there  must  be  a  strong  correlation 
between  cause  and  effect. 

•  The  difference  between  “levels”  and  “tutorials”  was  unclear.  At  what  point  does  the  player 
transition  from  one  “level”  to  another  versus  move  on  to  the  next  “tutorial?”  Consider 
explaining  the  difference  or  at  least  marking  the  transition  by  using  a  different  background  (or 
other  visual  or  auditory  cue)  to  provide  a  sense  of  progression. 

C.2.2  Match  between  the  System  and  the  Real  World 

The  game  should  speak  the  player ’s  language,  with  words,  phrases  and  concepts  familiar  to 
the  player,  rather  than  system-oriented  terms.  Follow  real-world  conventions,  making 
information  appear  in  a  natural  and  logical  order.  What  terms  are  out  of  place  or  poorly  aligned 
with  what  they  represent? 

•  The  scoring  mechanism  is  too  opaque;  there  does  not  seem  to  be  a  strong  correlation  between 
the  filter  options  chosen  and  the  number  of  points  scored.  Would  the  score  ever  reflect  the  fact 
that  a  solution  might  require  a  player  to  select  a  non-pink  filter  option?  Is  it  always  best  to 
select  the  pink  filters  (if  so,  why  bother  providing  players  with  other  options)? 

C.2.3  User  Control  and  Freedom 

Players  often  choose  actions  by  mistake  and  will  need  a  clearly  marked  “emergency  exit  ”  to 
leave  the  unwanted  state  without  having  to  go  through  an  extended  dialogue.  Support  undo  and 
redo.  What  mistakes  require  too  much  work  to  recover  from? 

•  Going  back  a  step  (to  a  previous  nucleus)  to  try  a  new  approach  is  a  very  helpful  ability; 
however,  if  a  user  decides  to  undo  their  undo  and  go  back  to  their  original  option,  the  player  is 
presented  with  completely  new  quark  filter  options  (which  may  not  include  the  desired  original 
option). 

C.2.4  Consistency  and  Standards 

Players  should  not  have  to  wonder  whether  different  words,  situations,  or  actions  mean  the 
same  thing.  Follow  platform  conventions.  What  inconsistencies  make  playing  or  understanding 
the  game  harder? 

•  Quark  filter  options  are  not  presented  in  a  consistent  manner.  Every  time  a  user  clicks  on  the 
nucleus,  they  are  provided  with  a  different  filter  option  set. 

C.2.5  Error  Prevention 

Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a  problem  from 
occurring  in  the  first  place.  Either  eliminate  error-prone  conditions  or  check  for  them  and 
present  players  with  a  confirmation  option  before  they  commit  to  the  action.  Does  the  game  let 
me  blunder  into  serious  errors? 

•  Did  not  seem  to  be  any  issues  during  the  evaluation. 

C.2.6  Recognition  rather  than  Recall 

Minimize  the  player ’s  memory  load  by  making  objects,  actions,  and  options  visible.  The 
player  should  not  have  to  remember  information  from  one  part  of  the  game  to  another. 
Instructions  for  use  of  the  system  should  be  visible  or  easily  retrievable  whenever  appropriate. 
Which  instructions  are  difficult  to  find  or  refer  to? 

•  The  game  does  a  good  job  of  simplifying  the  mechanic  of  dividing/separating  quarks,  which 
prevents  the  player  from  having  to  remember  any  critical  information. 
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C.2.7  Flexibility  and  Efficiency  of  Use 

Accelerators — unseen  by  the  novice  player — may  often  speed  up  the  interaction  for  the  expert 
player  so  that  the  game  can  cater  to  both  inexperienced  and  experienced  players.  Allow  players 
to  tailor  frequent  actions.  What  actions  do  I  find  repetitive  or  wish  I  could  hotkey/macro? 

•  The  manner  in  which  quark  filter  options  are  presented  requires  an  unnecessary  degree  of 
coordination  to  cycle  though  them. 

•  The  size  of  the  filter  options  requires  more  precision  than  necessary.  Making  the  target 
sizes  bigger  would  alleviate  the  precision  requirement  (see  Fitts’s  Law). 

•  People  are  more  effective  at  using  a  mouse  to  move  in  straight  lines.  The  circular  option 
presentation  format  requires  users  to  move  the  mouse  in  a  circular  motion,  which  requires 
amore  precision  and  time  than  a  linear  fonnat.  If  a  circular  presentation  ring  is  an  essential 
design  choice,  then  consider  making  the  option  ring  larger  with  larger  targets.  If  the  ring  is 
not  essential,  please  consider  a  linear  presentation  format. 

•  Please  consider  allowing  the  user  to  cycle  though  the  options  with  the  arrow  keys. 

C.2.8  Aesthetic  and  Minimalist  Design 

Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Every  extra 
unit  of  information  in  a  dialogue  competes  with  the  relevant  units  of  information  and  diminishes 
their  relative  visibility.  What  can  be  removed  to  improve  clarity? 

•  The  game  was  not  overloaded  with  distracting  graphical  elements.  For  the  most  part,  every 
graphical  element  had  a  purpose;  it  struck  a  decent  balance  with  the  data-ink  ratio. 

C.2.9  Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors 

Error  or  failure  messages  should  be  expressed  in  plain  language  (no  codes),  precisely 
indicate  the  problem,  and  constructively  suggest  a  solution.  When  I  fail  or  make  a  mistake,  do  I 
understand  why  and  how  to  do  better? 

•  There  did  not  seem  to  be  any  situations  where  a  player  would  be  unable  to  recover  from  a 
mistake  or  error,  other  than  when  a  desired  filter  option  disappears  from  the  ring  when  the 
player  accidentally  clicked  off  of  it  and  receives  a  new  filter  ring. 

C.2.10  Help  and  Documentation 

Even  though  it  is  better  if  the  game  can  be  used  without  documentation,  it  may  be  necessary 
to  provide  help  and  documentation.  This  information  should  be  easy  to  search,  list  concrete  steps 
to  be  carried  out,  and  not  be  too  large.  What  things  do  I  wish  I  could  look  up  quickly  while 
playing? 

•  There  did  not  appear  to  be  any  mechanism  for  viewing  help  documents  and  materials.  Although 
this  game  is  “simple,”  a  user  guide  might  prove  useful  to  explain  any  advanced  techniques. 

C.3  Game  Questions 
C.3.1  Tutorial  Analysis 

Assessment  of  the  effectiveness  of  the  tutorial;  how  effective  it  is  at  teaching  its  topics,  how 
well  you  feel  you  understand  the  game  at  the  end  of  it,  how  well  it  sets  you  up  for  the  jump  to 
real  gameplay,  how  well  it  draws  you  into  the  game.  Did  the  tutorial  help?  What  kind  of  player 
does  it  support?  Would  you  finish  it? 

•  The  tutorial  explains  the  concepts  well,  and  lets  the  player  get  some  experience  with  them. 

•  The  explanation  of  why  you  might  sometimes  want  to  take  a  filter  that  isn’t  the  brightest  is  not 
very  clear;  it  would  be  more  helpful  to  explicitly  explain  it. 
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•  This  is  also  true  of  the  filter  cap,  which  doesn’t  seem  to  be  explained. 

C.3.2  Engagement  Analysis 

Assessment  of  overall  enjoyment,  including:  what  elements  make  the  game  enjoyable,  what 
elements  detract.  How  can  elements  be  changed  to  enhance  enjoyment? 

•  It  is  satisfying  to  play  when  you  are  solving  the  puzzles,  but  eventually  gets  repetitive.  The 
unlocks  should  help  this  once  implemented,  as  would  a  progression  of  difficulty  (or  at  least 
size)  in  the  puzzles. 

•  Need  to  be  sure  that  the  unlocks  are  more  spaced  out,  though;  right  now  it  seems  like  there’s 
a  level-up  after  each  level  and  that  means  a  really  quick  pace  through  the  unlocks. 

C.3.3  Coherency  Analysis 

Assessment  of  coherency  and  balance  between  elements,  such  as:  mechanics,  story, 
aesthetics,  and  technology.  What  elements  seem  to  conflict  or  undermine  each  other? 

•  The  name,  aesthetics,  avatar  pictures,  and  game  mechanics  all  work  really  well  to  reinforce 
each  other. 

•  The  filter  coloring  should  convey  more  information  than  it  seems  to.  If  there  are  times  where 
it  is  good  to  split  into  two  groups  with  mixed  colors  and  other  times  where  it  is  good  to  split 
off  as  many  of  one  color  as  possible,  then  the  best  use  of  colors  would  be  to  communicate: 

•  Is  this  making  0,  1,  or  2  mixed-color  piles? 

•  Is  this  filter  making  the  best  split  of  its  type  among  the  filters  shown?  That  is,  dividing  the 
most  evenly  if  both  piles  are  mixed  color  vs.  separating  out  the  largest  single-color  pile  if 
there’s  only  1  mixed  pile. 

C.3.4  Analysis  of  Value  to  Player  Provided  by  Game 

What  is  the  player  getting  out  of playing  the  game?  Why  would  they  press  the  “ next  level" 
button?  Is  it  fun?  Are  they  making  progress?  Are  they  leveling  up  or  filling  a  bar  or 
something?  Are  their  game-playing  peers  depending  on  them? 

•  The  intrinsic  value  of  the  puzzle-solving  is  good.  A  solid  unlock  system  and  a  progression  of 
puzzle  size/difficulty/complexity/etc.  would  enhance  it  even  more. 

•  There  doesn’t  seem  to  be  any  hooks  for  extrinsic  motivation,  such  as  leaderboards,  the  overall 
progress  that  all  players  are  contributing  to,  etc.  The  “best”  benchmark  would  at  least  help 
them  judge  their  own  solutions  (once  there  are  others  to  compare  to),  but  some  indication  of 
overall  progress  both  individually  and  for  the  whole  group  would  seem  important  to  the  players 
you’re  looking  to  attract. 
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Appendix  D.  Expert  Game  Analysis:  Monsterproof 

D.1  Quick  Summary 

D.1.1  Best  Features 

•  The  graphics,  humor,  and  progression  are  all  quite  engaging  and  solving  a  puzzle  is  satisfying. 

•  The  theme  mostly  enhances  understanding  of  the  game  and  provides  a  good  framework  to 
explain  the  very  complex  topic. 

•  Setting  the  user  up  to  expect  a  game  based  on  logic  puzzles  right  from  the  start  makes  it  easier 
to  get  into  the  game  and  understand  the  motivation. 

D.1 .2  Needs  Improvement 

•  The  interface  has  several  clunky  parts  that  require  extra  clicks  to  accomplish  frequently 
repeated  tasks  and  doesn’t  always  give  you  a  good  sense  of  where  you  are  or  what  you’re  about 
to  change. 

•  The  complexity  ramps  up  quickly.  Even  though  the  tutorials  explain  well,  moving  directly  on 
to  the  next  tutorial  means  that  the  player  quickly  forgets  old  lessons  or  never  truly  grasps  the 
concepts  being  taught.  Each  coherent  group  of  tutorials  really  needs  some  form  of  “homework” 
that  follows  up  so  the  player  can  practice  what  was  learned.  Like  “ok,  now  you  try  -  go  gather 
3000  stones.” 


Table  9  Areas  of  Analysis  and  Priorities  for  the  Monsterproof  Game 


Area 

Priority 

Usability  questions:  Heuristics  analysis 

Visibility  of  system  status 

Medium 

Match  between  the  system  and  the  real  world 

Medium 

User  control  and  freedom 

Medium 

Consistency  and  standards 

Medium 

Error  prevention 

High 

Recognition  rather  than  recall 

High 

Flexibility  and  efficiency  of  use 

High 

Aesthetic  and  minimalist  design 

Low 

Help  users  recognize,  diagnose,  and  recover  from  errors 

Low 

Help  and  documentation 

Low 

Game  questions 

Tutorial  analysis 

High 

Engagement  analysis 

Low 

Coherency  analysis 

Low 

Analysis  of  value  to  player  provided  by  game 

High 

D.2  Usability  Questions:  Heuristics  Analysis 
D.2.1  Visibility  of  System  Status 

The  game  should  keep  players  informed  about  what  is  going  on,  through  appropriate 
feedback  within  reasonable  time.  What  feedback  is  missing? 

•  The  cards  for  operations  are  often  difficult  to  parse.  In  particular,  the  operation  symbols  could 
be  significantly  larger,  or  could  enlarge  when  they  are  selected. 
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•  The  visual  cue  for  which  scene  you  are  working  on  and  which  ones  are  locked  or  need  fixing 
(the  little  icons  on  the  buildings  along  the  path)  is  subtle  and  can  be  easy  to  miss.  It  is  also 
unclear  which  of  those  icons  corresponds  to  your  current  location. 

•  The  bets  interface  is  quite  hidden — you  can  place  a  bet,  but  you  can’t  see  whether  or  not  you’ve 
placed  one,  you  can’t  check  up  on  your  previous  bets  to  see  their  status,  and  the  messages  for 
winning  or  losing  a  bet  don’t  give  you  a  good  sense  of  what  you  bet  on.  It  would  also  be  nice 
to  go  back  to  a  level  once  you’ve  been  told  your  bet  failed. 

D.2.2  Match  between  the  System  and  the  Real  World 

The  game  should  speak  the  player ’s  language,  with  words,  phrases  and  concepts  familiar  to 
the  player,  rather  than  system-oriented  terms.  Follow  real-world  conventions,  making 
information  appear  in  a  natural  and  logical  order.  What  terms  are  out  of  place  or  poorly  aligned 
with  what  they  represent? 

•  The  analogy  of  the  gates  and  the  idea  of  keeping  monsters  out  have  been  changed,  so  the  idea 
that  you  are  “monsterproofing”  the  area  (especially  given  that  you  are  portrayed  as  a  monster 
yourself)  doesn’t  seem  to  jibe.  This  has  a  couple  of  consequences: 

•  “Open  Sesame”  as  the  submit  action  and  “gates,”  both  the  tenn  and  the  artwork,  don’t 
really  make  sense  to  the  player.  Instead,  potentially  just  mention  that  the  gates  being  open 
lets  you  know  the  road  is  accessible  or  something  similar. 

•  With  the  focus  being  on  creating  roads  (which  makes  sense  mechanically),  it  seems  like 
the  icons  for  accessing  the  different  sections  of  a  level  should  be  directly  on  a  road  piece. 
This  would  mean  creating  a  road  piece  that  went  “through”  the  forest  in  some  sense  (from 
the  forest  icon  to  the  gold  mine  icon  on  the  most  simple  forest  levels).  On  the  plus  side, 
these  would  be  easier  to  see. 

•  Could  the  activity  be  reframed  to  explicitly  be  about  getting  the  teams  of  monsters  from  each 
site  to  cooperate  on  building  a  road?  Then  “open  sesame”  can  just  be  “build  road,”  and  it  now 
makes  sense  that  you  are  trying  to  “say  the  same  thing”  but  in  terms  that  the  monsters  at  both 
sites  understand.  It  even  lends  itself  to  the  day-shift/night-shift  analogy  in  the  forest  (it’s  harder 
to  build  roads  through  forest  or  something  similar). 

•  Score  seems  like  a  thing  I  want  a  lot  of.  Most  games  in  which  you  are  trying  to  get  less  “score” 
tend  to  call  it  something  else,  like  “strokes”  or  “moves”  or  whatever.  Would  “cost”  or  “parts” 
or  something  like  that  work?  Cost  or  something  similar  would  also  make  “freebies”  very  clear. 

•  A  minor  thing,  but  Mon  Academy  can  just  be  Academy,  in  the  same  way  that  the  quarry  and 
the  sawmill  just  use  familiar  words. 

D.2.3  User  Control  and  Freedom 

Players  often  choose  actions  by  mistake  and  will  need  a  clearly  marked  "emergency  exit"  to 
leave  the  unwanted  state  without  having  to  go  through  an  extended  dialogue.  Support  undo  and 
redo.  What  mistakes  require  too  much  work  to  recover  from? 

•  Undo  is  crucial — it  is  often  difficult  to  drag  or  remove  the  exact  piece  you  want  and  it  is  easy 
to  do  something  that  is  then  difficult  to  manually  undo. 

•  A  reset  or  clear  button  to  set  you  back  to  the  original  state  is  needed,  especially  on  levels  where 
you  can  modify  top  and  bottom. 

•  A  save/load  capability  is  needed  so  that  you  can  return  to  a  puzzle  later  (not  just  when  you 
leave  for  the  day,  but  also  if  you  want  to  try  something  else  for  a  bit,  or  just  go  out  and  sell 
some  resources  so  you  can  bet  big  on  the  puzzle  being  unsolvable). 
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D.2.4  Consistency  and  Standards 

Players  should  not  have  to  wonder  whether  different  words,  situations,  or  actions  mean  the 
same  thing.  Follow  platform  conventions.  What  inconsistencies  make  playing  or  understanding 
the  game  harder? 

•  The  drag  and  drop  interface  is  mostly  intuitive,  but  the  card-based  backing  system  is  not. 
Despite  the  instructions  in  the  tutorial,  the  inclination  when  trying  to  form  “snake  less  than  or 
equal  to  dragon”  is  to  drag  the  snake  onto  the  field,  then  build  the  other  pieces  off  of  it,  left  to 
right,  like  you  would  write  it.  Starting  with  snake  less  than  or  equal  to  fairy,  then  replacing 
fairy  with  dragon  seems  like  an  odd  way  to  go  about  it,  even  if  it  is  fewer  steps. 

•  Score  only  seems  to  count  on  the  last  step,  which  creates  an  inconsistency  in  the  rule  about 
whether  or  not  to  prefer  freebies — basically,  if  everything  is  free  in  the  earlier  sections,  then  I 
should  be  willing  to  make  stuff  complex  to  leverage  freebies  and  simplify  the  last  step  as  much 
as  possible.  Perhaps  that’s  desirable  behavior,  but  it  seems  inconsistent/exploitative. 

•  Why  does  adding  or  subtracting  a  number  require  me  to  double-click  and  then  choose 
“add/subtract  zero,”  but  multiplying  or  dividing  means  dragging  a  new  card  in  from  the  quill 
menu?  Seems  like  I  should  just  be  able  to  choose  a  single  menu  item  to  add  an  operator  and  a 
different  menu  item  to  swap  an  operator  to  something  else. 

•  This  could  be  done  with  a  single  on-screen  location  with  a  “plus  zero”  icon  that  could  be 
swapped  to  “minus  zero”  or  “divide  by  1”  or  “multiply  by  1,”  etc. 

•  The  tray  for  freebies  looks  more  like  a  visual  element  than  a  tray — there  should  be  more  of  a 
cue  that  it  is  a  workspace  for  things  to  be  dragged  to  (like  the  two  on  the  left). 

•  Dragging  a  card  (such  as  a  piece  of  an  equation)  from  the  panels  on  the  left  allows  you  to  treat 
that  card  like  a  monster  (substitute  it  in,  or  add  to  both  sides,  etc.),  but  you  can’t  do  this  with 
freebies  or  their  subsections. 

•  It’s  odd  that  I  can  drag  a  monster  onto  a  slot  and  then  choose  “plus  zero.” 

D.2.5  Error  Prevention 

Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a  problem  from 
occurring  in  the  first  place.  Either  eliminate  error-prone  conditions  or  check  for  them  and 
present  players  with  a  confirmation  option  before  they  commit  to  the  action  .  Does  the  game  let 
me  blunder  into  serious  errors? 

•  It  is  fairly  easy  to  blunder  into  bad  situations  because  the  consequences  of  interface  actions 
are  not  always  clear,  and  because  there  is  no  undo  (perhaps  some  of  these  are  due  to  bugs): 

•  Problems  seem  to  occur  when  accidentally  dropping  freebies  into  the  wrong  place,  or 
dropping  monsters  or  other  cards  onto  freebies,  or  double-clicking  freebies  (this  might  have 
been  fixed  in  Chrome,  since  I  couldn’t  reproduce  this  error;  it  was  reproducible  in  Firefox 
though) 

•  Dropping  a  monster  onto  an  operation  card  can  have  unexpected  results  if  the  whole  card 
is  highlighted 

•  A  confirmation  that  I’d  like  to  substitute  when  that’s  the  only  option  would  be  more  useful 
if  it  showed  what  was  being  substituted  for  (like  perhaps  if  the  top  of  the  dialog  showed 
the  card  to  be  changed  or  replaced).  This  is  especially  true  for  “remove;”  it  would  be  nice 
to  confirm  what  it  is  I’m  removing  and  not  end  up  accidentally  deleting  the  whole  equation 
over  and  over  again. 
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•  It  would  be  worth  explaining  the  card-based  UI  a  little  more.  The  highlighting  is  quite  clear 
once  you  get  it,  but  not  necessarily  obvious  at  the  start  (and  a  potential  source  of  frustration  if 
you  don’t  understand  why  the  context  menu  keeps  changing). 

•  It  would  be  helpful  to  show  a  symbol  of  some  sort  when  the  user  is  hovering  over  an  action 
that  isn’t  allowed  (like  if  they  are  about  to  drop  a  monster  onto  the  field  without  adding  an 
equation  first,  or  about  to  drag  a  freebie  into  the  main  equation,  etc.). 

D.2.6  Recognition  rather  than  Recall 

Minimize  the  player ’s  memory  load  by  making  objects,  actions,  and  options  visible.  The 
player  should  not  have  to  remember  information  from  one  part  of  the  game  to  another. 
Instructions  for  use  of  the  system  should  be  visible  or  easily  retrievable  whenever  appropriate. 
Which  instructions  are  difficult  to  find  or  refer  to? 

•  The  video  tutorials  make  it  difficult  to  quickly  refer  back  to  instructions,  especially  if  they 
aren’t  in  the  tutorial  you  are  currently  working  on  (or  you’re  on  a  level  beyond  the  tutorials). 

•  Some  form  of  quick  reference  card  for  each  tutorial  would  be  very  helpful.  Like  showing 
a  “copy-up”  play  with  “Assume  the  conclusion!”  or  “Try  copy-up!”  as  the  card  header. 

•  This  also  applies  to  using  the  pop-up  menu — there  is  no  indication  of  what  you  are  acting  on 
so  you  have  to  remember. 

•  Collapsed  cards  can  require  some  of  this  as  well;  It’s  like  playing  memory  when  you  want  to 
remember  what  they  do,  or  else  you  have  to  scroll  around  a  huge  field  of  them  with  them  all 
opened  (and  the  equation  closes  itself  when  you  interact  with  it). 

D.2.7  Flexibility  and  Efficiency  of  Use 

Accelerators — unseen  by  the  novice  player — may  speed  up  the  interaction  for  the  expert 
player  so  that  the  game  can  cater  to  both  inexperienced  and  experienced  players.  Allow  players 
to  tailor  frequent  actions.  What  actions  do  I find  repetitive  or  wish  I  could  hotkey/macro? 

•  Most  frequent  actions:  expand  all  sub-parts  of  a  card,  search  for  a  monster,  substitute  one  card 
for  another,  add  a  tenn  to  one  side,  simplify  an  expression,  and  remove  a  tenn  from  one  side. 
Negate  a  tenn  would  also  be  one,  but  it  isn’t  clear  if  that  can  be  done  beyond  using  the  simplify 
command. 

•  Of  these,  searching  for  a  monster  and  simplifying  are  relatively  easy. 

•  Substituting  one  card  for  another  and  adding  a  tenn  to  one  side  have  multiple  steps  that 
slow  them  down. 

•  Remove  a  term  from  one  side  is  difficult  and  dangerous  since  you  might  remove  the  whole 
equation  (I  mostly  subbed  in  the  appropriate  identity  and  then  simplify.) 

•  Expand  all  sub-parts  requires  individual  clicks  to  drill  down  to  them;  a  single  command 
would  be  better 

•  The  incredible  complexity  of  some  of  the  equations  can  be  overwhelming,  especially  early  on. 
Creating  a  system  by  which  a  user  can  substitute  a  single  icon  to  represent  a  complex 
expression  everywhere  it  shows  up  (like  using  HornedBull  instead  of 
B  [Bull  [0  [HornedSkull->l  ]  ]  ]  )  would  help  simplify  the  thought  process  of  dealing  with 
a  puzzle.  It  might  not  always  be  useful  (you  might  need  to  understand  the  guts  to  find  the  right 
freebie),  but  it  could  at  least  help  you  realize  that  the  simple  solutions  won’t  work  and  find  the 
puzzles  where  it’s  necessary  to  drill  down. 
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•  This  could  also  be  done  automatically  to  simplify  levels  for  novice  users.  They  might  not 
be  able  to  solve  the  simpler  level  (so  they  could  place  a  bet  on  that),  but  it  would  still  keep 
them  from  hitting  that  massive  bump  up  after  the  tutorial. 

•  If  save  and  load  is  implemented,  then  a  notes  feature  that  lets  you  tell  yourself  where  you  were 
at  on  a  puzzle  when  you  reload  would  be  nice. 

D.2.8  Aesthetic  and  Minimalist  Design 

Dialogs  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Every  extra  unit 
of  information  in  a  dialog  competes  with  the  relevant  units  of  information  and  diminishes  their 
relative  visibility.  What  can  be  removed  to  improve  clarity? 

•  This  is  all  much  improved  with  the  chests  and  the  easy  search  for  monsters  and  filtering  of  the 
freebies,  although  the  freebies  monsters  could  use  their  own  chest-like  groupings  of  “included 
in  the  answer”  and  “not  included  in  the  answer”  at  the  very  least  (or  just  sort  the  list  so  the  ones 
in  the  answer  are  always  at  the  front). 

•  Can  the  chests  with  0  in  them  be  removed?  If  not,  can  all  the  chests  that  don’t  contain  a 
monster  that  is  currently  in  the  equation  be  hidden  by  default  (and  only  shown  through  a 
user  command)? 

•  Using  the  substitution  system  described  above  to  simplify  equations  initially  (perhaps  letting 
advanced  players  try  to  “investigate”  a  monster  and  discover  its  inner  details)  might  make  for 
a  cleaner  presentation  than  the  list  of  N  monsters  that  many  puzzle  equations  collapse  to. 

D.2.9  Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors 

Error  or  failure  messages  should  be  expressed  in  plain  language  (no  codes),  precisely 
indicate  the  problem,  and  constructively  suggest  a  solution.  When  I  fail  or  make  a  mistake,  do  I 
understand  why  and  how  to  do  better? 

•  If  grabby  claws  prevent  you  from  winning,  they  should  cause  the  “Open  Sesame”  to  fail 

•  Some  form  of  quick  compare  for  the  complex  expressions  would  be  helpful,  like  selecting  two 
collapsed  cards  and  having  the  system  check  that  their  guts  are  the  same. 

•  The  idea  that  a  puzzle  might  not  be  solvable  should  be  emphasized  more  to  help  users  that  are 
really  struggling  find  a  way  out.  The  bets  interface  explanation  in  the  tutorial  is  fine,  but  easy 
to  forget  amongst  all  the  other  information  and  the  gavel  is  not  obviously  associated  with 
unsolvable  levels.  There  should  be  some  on-screen  text  or  other  reminder  that  when  you’re 
frustrated  with  a  level  it  might  be  because  there  actually  isn’t  an  answer. 

•  Save/load  would  also  help  here,  since  you  could  just  pause  and  try  again  later  when  you 
are  fresh. 

D.2.10  Help  and  Documentation 

Even  though  it  is  better  if  the  game  can  be  used  without  documentation,  it  may  be  necessary 
to  provide  help  and  documentation.  This  information  should  be  easy  to  search,  list  concrete  steps 
to  be  carried  out,  and  not  be  too  large.  What  things  do  I  wish  I  could  look  up  quickly  while 
playing? 

•  A  list  of  the  key  concepts  and/or  the  quick  reference  cards  mentioned  above  would  help. 

D.3  Game  Questions 
D.3.1  Tutorial  Analysis 

Assessment  of  the  effectiveness  of  the  tutorial;  how  effective  it  is  at  teaching  its  topics,  how 
well  you  feel  you  understand  the  game  at  the  end  of  it,  how  well  it  sets  you  up  for  the  jump  to 
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real  gameplay,  how  well  it  draws  you  into  the  game.  Did  the  tutorial  help?  What  kind  of  player 

does  it  support?  Would  you  finish  it? 

•  The  videos  explain  their  concepts  clearly  and  are  engaging  to  watch,  making  the  process  of 
learning  the  basics  enjoyable. 

•  The  clear  framing  of  the  game  as  a  logic  puzzle  right  from  the  start  does  a  great  job  of 
setting  expectations  and  helping  the  user  engage  with  the  game. 

•  Using  only  videos  isn’t  ideal  because  they  require  a  lot  of  attention,  memory,  and  headphones. 
Some  of  this  would  be  mitigated  if  there  was  more  text  to  help  identify  what’s  happening  (even 
very  minimalistic  text). 

•  Some  form  of  tutorial  summary  reference  card  that  you  can  access  would  also  be  nice, 
rather  than  hunting  through  old  tutorials  to  find  an  explanation. 

•  Another  drawback  to  video-based  tutorials  is  that  they  don’t  age  well,  UI,  art,  etc.  that  changes 
in  the  game  doesn’t  change  in  the  video,  so  they  get  stale  and  potentially  misleading  or  they 
need  to  be  updated. 

•  Specific  comments  on  the  tutorials: 

•  Electives  are  poorly  placed — it  seems  like  they  should  be  in  a  separate  section  that  can  be 
referred  to  as  needed,  or  they  should  be  required.  It  also  seems  like  the  elective  that  unlocks 
the  workshop  isn’t  really  optional. 

•  Obvious  stamps  on  completed  tutorials  and  a  scrollbar  showing  where  you  are  in  the  list 
would  make  it  clearer  what  you’ve  done  and  how  far  you  have  to  go. 

•  Sound  levels  are  different  for  different  tutorials — in  particular,  professor  Voiron’s  voice  is 
harder  to  hear  than  Joey’s. 

•  In  the  section  that  discusses  buying  things,  it  would  be  better  to  start  the  list  off  with  some 
things  that  are  real  and  then  drop  in  something  that  is  obviously  a  joke.  As  it  is,  the  list  of 
all  joke  things  makes  it  seem  like  some  of  those  things  are  real  things  you  can  buy. 

•  The  bump  from  tutorial  to  gameplay  remains  problematic.  The  elements  of  a  great,  smooth 
transition  are  there  if  each  new  concept  (combining  an  equation  with  a  freebie,  modifying  an 
equation  to  make  the  next  step  work,  wobbles,  forests,  each  complex  operator,  grabby  claws, 
etc.)  was  rolled  out  with  some  tutorials  followed  by  maybe  three  levels  that  practice  it  (and 
perhaps  also  use  previously  introduced  concepts  as  needed)  before  the  next  concept  and  its 
corresponding  content  are  unlocked. 

•  Rolling  up  complications  that  have  not  yet  been  introduced  so  that  the  player  can’t 
manipulate  them  would  allow  this  to  operate  on  real  levels  during  the  practice  sessions, 
with  the  potential  cost  of  having  unsolvable  levels. 

•  Alternately,  these  practice  levels  could  be  curated  or  crafted.  While  that  might  produce  a 
longer  lead  time  from  when  a  player  starts  to  when  they  are  solving  real  levels,  players  that 
are  overwhelmed  by  complexity  are  unlikely  to  solve  anything  either. 

D.3.2  Engagement  Analysis 

Assessment  of  overall  enjoyment,  including:  what  elements  make  the  game  enjoyable,  what 

elements  detract.  How  can  elements  be  changed  to  enhance  enjoyment? 

•  Solving  a  puzzle  is  very  satisfying.  Even  just  being  hot  on  the  trail  of  a  possible  solution  is 
satisfying. 

•  Other  current  detractors: 
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•  Confusion  about  mechanics,  the  things  represented  in  equations,  and  the  options  for 
manipulating  expressions. 

•  Bugs  encountered: 

•  Lots  of  graphical  lag  on  one  of  our  machines  right  from  the  start  with  a  new  browser 
page  (so  this  was  not  related  to  a  memory  leak). 

•  Seemed  like  a  level  was  solved,  but  Open  Sesame  didn’t  work  and  it  wasn’t  clear  why 
not. 

•  Bugs  with  dragging  cards  to  places  they  weren’t  supposed  to  be  (seems  like  this  can 
still  occur  if  you  substitute  a  single  monster  for  the  entire  equation). 

•  Music  during  videos  is  too  loud. 

•  Could  sometimes  enter  a  mode  like  you  were  editing  a  freebie. 

•  Making  a  mistake  and  having  to  redo  a  significant  amount  of  work  is  a  big  detractor. 

•  The  unlock  tree,  the  buildings  that  pay  out  resources  when  you  return,  and  the  betting  mechanic 
seem  like  good  cross-session  engagement  factors. 

D.3.3  Coherency  Analysis 

Assessment  of  coherency  and  balance  between  elements,  such  as:  mechanics,  story, 
aesthetics,  and  technology.  What  elements  seem  to  conflict  or  undermine  each  other? 

•  The  concept  of  gates  you  are  opening  doesn’t  seem  to  be  supported  by  the  other  elements  of 
the  story. 

D.3.4  Analysis  of  Value  to  Player  Provided  by  Game 

What  is  the  player  getting  out  of playing  the  game?  Why  would  they  press  the  “ next  level’’ 
button?  Is  it  fun?  Are  they  making  progress?  Are  they  leveling  up  or  filling  a  bar  or 
something?  Are  their  game-playing  peers  depending  on  them? 

•  The  intrinsic  value  of  the  puzzle  solving  is  quite  good,  and  the  smoother  the  interface  can  be 
(both  for  the  puzzle  solving  and  the  betting)  the  more  fun  it  will  be  to  just  sit  and  work  on 
puzzles. 

•  There  is  also  some  value  provided  by  curiosity  at  unlocking  new  buildings  though  the  current 
display  (which  doesn’t  show  that  there’s  an  unlock  tree  until  you  unlock  a  pre-requisite)  and 
the  small  number  of  buildings  on  the  main  screen  make  it  clear  that  there  is  not  much  to  unlock. 
This  would  all  be  more  of  a  motivation  if  more  of  the  mechanics  were  hidden  and  only 
unlocked  through  purchases  at  the  upgrade  shop. 

•  A  Daily  (Weekly?)  Puzzle  or  something  similar  would  give  players  more  value  for  returning 
to  play — perhaps  with  bonus  prizes  for  the  first  person  to  solve  it  or  the  first  N  people  to  solve 
it.  Could  make  the  refresh  time  known  so  that  competitive  players  can  anticipate  it. 

•  A  leaderboard  or  similar  concept  for  puzzle  scores,  gold  spent/resources  gathered,  bets  won, 
etc.,  would  also  give  some  of  that  extrinsic  value  to  doing  well  at  the  game. 

•  Bugged  Level?  If  1+S  <=  A[B]  as  the  freebie  claims,  then  asserting  that  X  +  A[B]<=  QRS 
should  prove  that  X  +  1  +  S  <=  QRS 
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Appendix  E.  Expert  Game  Analysis:  Paradox 

E.1  Quick  Summary 

E.1.1  Best  Features 

•  The  autosolver  provides  a  smoother,  more  satisfying  way  to  play. 

•  Painting  with  the  brushes  makes  for  a  friendlier  method  of  interacting  with  the  game  elements. 

•  It  seems  like  all  the  provided  levels  can  actually  be  solved,  which  is  much  more  satisfying  to 
the  player. 

E.1. 2  Needs  Improvement 

•  The  tutorial  is  confusing  and  starts  off  with  many  under-the-hood  details  that  aren’t  needed.  It 
should  be  reworked  to  start  from  the  autosolver  and  only  get  the  other  information  to  the  player 
once  they  are  an  expert. 

•  Terms  and  symbols  are  difficult  to  understand  and  are  inconsistent. 

•  There  needs  to  be  some  additional  form  of  reward  loop,  level-up  progress,  or  other  value 
mechanism  to  keep  players  engaged  and  give  them  a  reason  to  play. 


Table  10  Areas  of  Analysis  and  Priorities  for  the  Paradox  Game 


Area 

Priority 

Usability  questions:  Heuristics  analysis 

Visibility  of  system  status 

Medium 

Match  between  the  system  and  the  real  world 

High 

User  control  and  freedom 

Low 

Consistency  and  standards 

Medium 

Error  prevention 

Low 

Recognition  rather  than  recall 

Low 

Flexibility  and  efficiency  of  use 

Low 

Aesthetic  and  minimalist  design 

Medium 

Help  users  recognize,  diagnose,  and  recover  from  errors 

Low 

Help  and  documentation 

Medium 

Game  questions 

Tutorial  analysis 

High 

Engagement  analysis 

High 

Coherency  analysis 

Medium 

Analysis  of  value  to  player  provided  by  game 

High 

E.2  Usability  Questions:  Heuristics  Analysis 
E.2.1  Visibility  of  System  Status 

The  game  should  keep  players  informed  about  what  is  going  on,  through  appropriate 
feedback  within  reasonable  time.  What  feedback  is  missing? 

•  Red  conflicts  are  frequently  difficult  to  find  on  the  playing  field  and  mini-map  because  they 
are  shown  behind  the  blue  circles.  As  I  drag,  the  autosolver  brush  circles  are  highlighted 
yellow.  When  the  autosolver  finishes,  these  circles  remain  yellow,  which  makes  it  seem  like 
they  are  still  being  autosolved.  It  would  be  more  helpful  as  feedback  if  they  changed  to  some 
other  color  (green?)  and  stayed  that  way  until  the  solver  starts  again  (since  it  is  helpful  to  know 
where  the  previously  solved  region  was) 
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•  It  is  difficult  to  determine  the  current  zoom  level,  making  it  difficult  to  determine  where  the 
player  is  in  relation  to  the  entire  puzzle.  This  problem  is  exacerbated  by  the  fact  that  the  playing 
field  is  redrawn  after  every  magnification  change. 

E.2.2  Match  between  the  System  and  the  Real  World 

The  game  should  speak  the  player ’s  language,  with  words,  phrases  and  concepts  familiar  to 
the  player,  rather  than  system-oriented  terms.  Follow  real-world  conventions,  making 
information  appear  in  a  natural  and  logical  order.  What  terms  are  out  of  place  or  poorly  aligned 
with  what  they  represent? 

•  The  names  “para”  and  “dox”  don’t  mean  anything  and  don’t  help  the  player  attach  ideas  or 
game  mechanics  to  them.  I  can’t  even  remember  which  thing  is  represented  by  which  name 
without  going  back  to  look  at  the  first  tutorial.  It  might  be  better  to  use  terms  like  “switch”  and 
“gate.” 

•  “Painting”  is  a  good  tenn  for  what  the  user  is  asked  to  do,  but  the  big  circle  and  the  hex,  square, 
and  circle  symbols  don’t  immediately  convey  the  idea  of  a  paintbrush  so  working  in  a 
paintbrush  graphic  to  the  list  of  symbols  and  perhaps  to  the  big  circle  itself  might  make  this 
clearer. 

•  Autosolver  is  a  clear  term,  but  has  the  downside  of  conveying  the  idea  that  the  computer  can 
do  this  work  itself,  which  immediately  brings  up  the  question  of  why  the  player  is  even 
involved.  This  is  exacerbated  by  the  autosolver  limit,  which  sounds  like  an  arbitrary  restriction 
on  your  autosolving  that  is  the  only  thing  preventing  you  from  getting  the  whole  answer  by 
just  highlighting  the  whole  map.  More  on  this  in  the  gameplay  area. 

•  It  would  be  better  to  use  terms  like  a  correction  brush  that  you  want  to  use  to  fix  flaws,  but 
which  can  run  out  of  ink. 

•  It  is  difficult  for  users  to  develop  a  model  of  the  game  mechanics  because  of  confusing  game 
instructions  and  loosely  coupled  feedback  loops  in  the  mechanics. 

E.2.3  User  Control  and  Freedom 

Players  often  choose  actions  by  mistake  and  will  need  a  clearly  marked  “emergency  exit’’  to 
leave  the  unwanted  state  without  having  to  go  through  an  extended  dialogue.  Support  undo  and 
redo.  What  mistakes  require  too  much  work  to  recover  from? 

•  Most  actions  can  be  undone  automatically,  but  if  you  try  the  autosolver  on  a  section  and  it 
breaks  things  more  than  it  helps,  it  can  be  a  pain  to  get  back  to  where  you  were.  It  would  be 
nice  to  be  able  to  undo  the  previous  autosolve  attempt  at  least. 

E.2.4  Consistency  and  Standards 

Players  should  not  have  to  wonder  whether  different  words,  situations,  or  actions  mean  the 
same  thing.  Follow  platform  conventions.  What  inconsistencies  make  playing  or  understanding 
the  game  harder? 

•  In  terms  of  platform  conventions,  being  able  to  drag  the  playing  area  around  with  the  mouse 
would  be  very  helpful;  the  arrows  keys/mouse  to  edge  of  screen  are  somewhat  clunky  and 
dated  navigation  tools. 

•  The  zoom  levels  are  also  not  very  granular  and  somewhat  clunky  to  apply;  mousewheel 
zoom  is  the  expected  default  here. 

•  The  icons  used  to  represent  the  different  states  of  the  paras  don’t  correspond  to  the  symbols  on 
the  brushes  that  paint  them,  making  it  difficult  to  distinguish  how  each  brush  affects  the  paras. 
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•  The  color-coding  of  the  lines  between  dox  and  para  are  inconsistent;  sometimes  it  seems  like 
they  are  showing  whether  there’s  a  match  or  not  and  sometimes  it  seems  like  they  are  just 
based  on  one  of  the  two  colors.  Would  be  best  if  they  just  showed  orange  when  the  endpoints 
didn’t  match  and  gray  when  they  did. 

•  Some  conflicts  seemed  to  be  resolved  by  painting  the  same  area  multiple  times.  This  seems 
inconsistent  to  the  user.  Even  if  this  is  because  the  user  painted  the  area  in  the  correct  order  to 
resolve  the  conflict,  there  can  be  a  sense  of  inconsistency  to  the  action.  The  brush  is  so  big  and 
the  paras/doxes  are  so  tiny,  the  player  cannot  easily  see  what  has  happened  to  differentiate 
between  the  correct  resolution  action  and  the  failed  resolution  action.  The  relationship  between 
the  large,  autosolving  brush  and  the  small,  numerous  para/doxes,  makes  for  an  impoverished 
feedback  loop  and  hinders  players  from  learning  the  game  mechanics  and  how  to  prevent  new 
conflicts. 

E.2.5  Error  Prevention 

Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a  problem  from 
occurring  in  the  first  place.  Either  eliminate  error-prone  conditions  or  check  for  them  and 
present  players  with  a  confirmation  option  before  they  commit  to  the  action.  Does  the  game  let 
me  blunder  into  serious  errors? 

•  All  of  the  game  states  are  reachable  from  each  other  so  serious  errors  are  not  truly  possible 

E.2.6  Recognition  rather  than  Recall 

Minimize  the  player ’s  memory  load  by  making  objects,  actions,  and  options  visible.  The 
player  should  not  have  to  remember  information  from  one  part  of  the  game  to  another. 
Instructions  for  use  of  the  system  should  be  visible  or  easily  retrievable  whenever  appropriate. 
Which  instructions  are  difficult  to  find  or  refer  to? 

•  All  necessary  information  is  displayed,  though  the  issues  with  the  correspondence  between 
symbols  (e.g.,  brushes)  and  their  meaning  make  it  harder  to  use. 

E.2.7  Flexibility  and  Efficiency  of  Use 

Accelerators — unseen  by  the  novice  player — can  speed  up  the  interaction  for  the  expert 
player  so  that  the  game  can  cater  to  both  inexperienced  and  experienced  players.  Allow’  players 
to  tailor  frequent  actions.  What  actions  do  I find  repetitive  or  wish  I  could  hotkey/macro? 

•  There  are  very  few  standard  actions  due  to  the  nature  of  the  game. 

E.2.8  Aesthetic  and  Minimalist  Design 

Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Every  extra 
unit  of  information  in  a  dialogue  competes  with  the  relevant  units  of  information  and  diminishes 
their  relative  visibility.  What  can  be  removed  to  improve  clarity? 

•  The  entire  explanation  of  paras  and  dox  and  states  and  painting  states  only  confuses  the  player 
and  makes  it  difficult  to  understand  the  goal.  More  on  this  in  the  Tutorial  Analysis  section. 

•  The  single-state  paintbrushes  seem  like  expert-level  tools  that  are  best  left  for  a  much  later 
unlock. 

•  For  that  matter,  the  actual  colors  of  the  nodes  and  the  dox  are  only  needed  when  you  are  using 
a  single-state  paintbrush. 

E.2.9  Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors 

Error  or  failure  messages  should  be  expressed  in  plain  language  (no  codes),  precisely 
indicate  the  problem,  and  constructively  suggest  a  solution.  When  I  fail  or  make  a  mistake,  do  I 
understand  why  and  how  to  do  better? 
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•  This  is  a  bit  of  a  problem  with  no  great  solution  because  it  wouldn’t  really  be  possible  to 
explain  how  to  do  things  correctly  with  the  types  of  errors  that  are  possible.  If  you  understand 
how  the  tools  work  you  know  all  there  is  to  know  about  why  they  fail. 

E.2.10  Help  and  Documentation 

Even  though  it  is  better  if  the  game  can  be  used  without  documentation,  it  may  be  necessary 
to  provide  help  and  documentation.  This  information  should  be  easy  to  search,  list  concrete  steps 
to  be  carried  out,  and  not  be  too  large.  What  things  do  I  wish  I  could  look  up  quickly  while 
playing? 

•  Desirable  help  sections:  a  list  of  controls,  an  explanation  of  the  brushes,  an  explanation  of 
states  and  conflicts,  maybe  tips  and  tricks  for  specific  situations  that  occur  frequently. 

•  Users  should  be  able  to  access  instructions,  such  as  an  explanation  of  the  states  and  how 
conflicts  arise  and  how  to  remove  them. 

E.3  Game  Questions 
E.3.1  Tutorial  Analysis 

Assessment  of  the  effectiveness  of  the  tutorial;  how  effective  it  is  at  teaching  its  topics,  how 
well  you  feel  you  understand  the  game  at  the  end  of  it,  how  well  it  sets  you  up  for  the  jump  to 
real  gameplay,  how  well  it  draws  you  into  the  game.  Did  the  tutorial  help?  What  kind  of  player 
does  it  support?  Would  you  finish  it? 

•  The  tutorial  leaves  the  new  user  quite  confused  about  the  mechanics  and  the  goal  of  the  game. 

•  The  tutorial  should  really  lead  off  with  the  goal  of  the  game,  giving  the  player  some  sense  of 
who  they  are,  what  they  are  doing,  and  why  they  are  doing  it.  The  most  important  thing  to 
emphasize  here  is  that  it  is  the  player’s  job  to  direct  resources  at  solving  sections  most  likely 
to  remove  the  conflicts  while  saving  resources  by  leaving  out  the  sections  that  don’t  need  to 
be  included. 

•  As  currently  constructed,  the  tutorial  starts  off  too  fiddly  and  loses  players  before  it  is  able  to 
hook  them  and  let  them  build  a  framework  for  what  they  are  doing.  The  analogy  that  comes  to 
mind  is  starting  off  your  tutorial  for  a  medieval  game  by  explaining  the  details  of  annor  class, 
attack  roll,  dodging,  weapon  type,  flanking  rules,  etc.,  when  all  the  player  really  needs  to  know 
is  when  to  attack  and  what  button  to  press  to  do  it. 

•  Suggest  reworking  the  tutorial  as  follows: 

•  Begin  with  the  autosolver  but  renamed  to  something  that  sounds  more  engaging  like  “the 
master  brush”  or  at  least  more  inclusive  of  the  player,  like  “optimizer.” 

•  Explain  that  your  goal  is  to  eliminate  conflicts  by  directing  the  autosolver,  then  let  the 
player  do  that  by  highlighting  the  entirety  of  a  small  level. 

•  Then  explain  the  limit  (e.g.,  by  saying  that  ink  is  limited  and  it  drains  out  when  you  paint). 

•  Then  let  the  player  play  with  this  in  a  bunch  of  levels  of  increasing  complexity,  gradually 
upping  the  limit. 

•  Once  the  player  has  gotten  through  some  complex  levels,  unlock  a  “show  states”  map  mode 
and  explain  how  conflicts  arise  when  the  states  are  mis-matched  and  let  the  player  see  how 
on  a  simple  level  the  autosolver  resolves  all  these  conflicts. 

•  Then  unlock  the  single-state  tools  and  explain  how  the  player  could  use  them  to  manually 
solve  or  shake  up  an  area  to  help  find  the  solution.  This  could  even  wait  until  the  user  is 
quite  experienced. 
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E.3.2  Engagement  Analysis 

Assessment  of  overall  enjoyment,  including  what  elements  make  the  game  enjoyable  and 
what  elements  detract.  How  can  elements  be  changed  to  enhance  enjoyment? 

•  Painting  areas  is  satisfying  and  enjoyable  when  it  is  working.  Making  progress  towards  full 
completion  is  also  satisfying,  even  at  the  very  end  when  I’m  trying  to  just  get  that  last  little  bit. 

•  Finding  the  conflicts  is  too  difficult  sometimes,  making  it  harder  to  play. 

•  Selecting  the  proper  things  can  be  tough  too;  zooming  in/out  and  panning  is  too  clunky  to  play 
on  all  but  fully  zoomed  out. 

•  Strategies  are  opaque — it  would  be  nice  to  get  some  fonn  of  reward  loop  going  for  succeeding 
at  finding  a  strategy. 

E.3.3  Coherency  Analysis 

Assessment  of  coherency  and  balance  between  elements,  such  as  mechanics,  story, 
aesthetics,  and  technology.  What  elements  seem  to  conflict  or  undermine  each  other? 

•  There  is  a  lack  of  coherence  in  the  visual  representation  of  the  different  elements — the  dox, 
the  paras,  the  conflicts,  the  symbols  for  the  brushes,  the  autosolver  limit,  etc. 

•  There  is  an  overemphasis  on  the  under-the-hood  details  for  a  game  that  is  really  about 
managing  autosolver  tasking. 

•  There  is  no  clear  story  or  goal  to  drive  the  player  beyond  the  idea  that  levels  should  be 
completed — more  of  a  sense  of  the  player’s  role  and  how  far  they’ve  gotten  would  help  bring 
the  elements  together. 

E.3.4  Analysis  of  Value  to  Player  Provided  by  Game 

What  is  the  player  getting  out  of  playing  the  game?  Why  would  they  press  the  “ next  level  ” 
button?  Is  it  fun?  Are  they  making  progress?  Are  they  leveling  up  or  filling  a  bar  or 
something?  Are  their  game-playing  peers  depending  on  them? 

•  Definitely  need  a  stronger  reward  loop;  increasing  the  autosolver  limit  gives  some  basic 
progression,  but  some  additional  fonn  of  reward  is  needed  for  there  to  be  an  intrinsic 
motivation  to  play. 

•  One  example  might  be  to  have  “extra  ink”  where  the  player  could  choose  to  overfill  the 
brush  (increasing  the  autosolver  limit)  a  certain  number  of  times  per  level  to  try  to  solve  a 
section  they  are  having  trouble  with.  This  might  cost  points  and  therefore  slow  the  player’s 
unlocking  of  more  extra  ink,  but  it  provides  a  cool  strategic  ability  that  the  player  has  an 
incentive  to  improve. 

•  The  extrinsic  reward  loop  is  also  not  clear  at  this  point — there’s  no  indication  of  scientific 
progress  being  made  or  of  how  the  community  is  doing  as  a  whole,  and  there’s  so  little 
introduction  or  grounding  in  why  it  is  valuable  to  be  doing  this  that  it’s  hard  to  be  motivated 
to  play  by  those  factors. 
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Appendix  F.  Expert  Game  Analysis:  Ghost  Map — Hyperspace 

F.1  Quick  Summary 

F.1.1  Best  Features 

•  Organizing  nodes  is  fun  to  do,  fighting  the  aliens  is  a  good  distraction  while  rifts  are  sealing, 
and  the  challenge  of  finding  a  good  spot  to  seal  can  be  fun  as  well  (when  it  works!). 

•  Love  the  smoky  intro  voice-over. 

F.1 .2  Needs  Improvement 

•  Better  indications  of  the  potential  strength  and  duration  of  a  seal  attempt  would  help  the  player 
leam  and  optimize  strategy. 

•  Closing  the  loop  on  various  mechanics  so  that  they  feed  back  into  each  other  would  increase 
the  engagement  value  and  the  potential  for  new  experiences  as  the  player  progresses. 


Table  11  Areas  of  Analysis  and  Priorities  for  the  Paradox  Game 


Area 

Priority 

Usability  questions:  Heuristics  analysis 

Visibility  of  system  status 

High 

Match  between  the  system  and  the  real  world 

High 

User  control  and  freedom 

Low 

Consistency  and  standards 

Medium 

Error  prevention 

Low 

Recognition  rather  than  recall 

Low 

Flexibility  and  efficiency  of  use 

Medium 

Aesthetic  and  minimalist  design 

Low 

Help  users  recognize,  diagnose,  and  recover  from  errors 

Medium 

Help  and  documentation 

Medium 

Game  questions 

Tutorial  analysis 

Medium 

Engagement  analysis 

High 

Coherency  analysis 

Low 

Analysis  of  value  to  player  provided  by  game 

High 

F.2  Usability  Questions:  Heuristics  Analysis 
F.2.1  Visibility  of  System  Status 

The  game  should  keep  players  informed  about  what  is  going  on,  through  appropriate 
feedback  within  a  reasonable  time  period.  What  feedback  is  missing? 

•  It  was  unclear  why  some  rift  sealing  attempts  were  unsuccessful,  despite  having  followed  the 
instructions. 

•  The  rift  sealing  in  general  would  be  much  clearer  if  there  was  a  meter  showing  you  some 
form  of  relative  strength  of  your  attempt.  Like  a  password  strength  meter,  but  applying  the 
rules  like  “include  2  fracture  points”  and  “follow  a  single  energy  signature.” 

•  It  could  be  made  clear  that  even  a  max  strength  attempt  might  fail,  and  this  would  help 
reinforce  that  idea  and  avoid  frustration  when  it  does. 
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•  It  would  also  help  to  have  a  meter  that  showed  a  relative  estimate  of  how  long  the  sealing 
would  take.  Even  if  this  is  only  a  guess  based  on  number  of  nodes  included,  showing  the 
user  that  adding  one  node  can  double  or  triple  the  time  it  takes  (or  more)  would  help  them 
make  the  tradeoffs. 

•  In  both  cases,  showing  the  estimates  for  the  current  selection  before  kicking  off  the  sealers 
would  help  reinforce  the  strategies  by  giving  the  player  positive  feedback  for  successfully 
finding  a  good  spot,  even  if  that  spot  turns  out  not  to  work. 

•  The  transition  to  combat  mode  is  unclear — some  visual  indicator  to  distinguish  combat  mode 
from  planning  mode  would  be  useful. 

•  The  explosion  feedback  for  aliens  multiplying  is  too  similar  to  the  explosion  for  destroying  an 
alien,  making  it  very  confusing  what  is  going  on  during  combat. 

•  It  isn’t  clear  that  exiting  a  level  will  allow  that  level  to  keep  calculating. 

F.2.2  Match  between  System  and  the  Real  World 

The  game  should  speak  the  player ’s  language,  with  words,  phrases,  and  concepts  familiar  to 
the  player,  rather  than  system-oriented  terms.  Follow  real-world  conventions,  making 
information  appear  in  a  natural  and  logical  order.  What  terms  are  out  of  place  or  poorly  aligned 
with  what  they  represent? 

•  It  isn’t  clear  what  the  goal  of  the  alien  attacks  is  and  therefore  what  the  purpose  of  defense 
against  them  is.  If  everything  is  about  the  rift  sealing,  more  connection  between  combat  and 
the  success  of  the  sealers  seems  needed. 

•  This  could  operate  as  a  soft  cap  on  the  amount  of  time  sealers  can  be  active;  eventually  you 
will  get  overwhelmed  if  the  sealing  hasn’t  succeeded. 

•  Could  also  be  a  good  hook  for  progression — better  weapons,  better  armor  or  shields  for  the 
sealers,  better  auto-pilot  combat,  etc. 

•  “Planets”  seems  like  an  odd  term  for  nodes  that  can  be  moved  around  and  that  the  player  is 
organizing.  Perhaps  this  would  be  better  phrased  as  a  defense  grid  or  hyperspace  grid  or 
something,  with  a  bunch  of  satellites  in  it  that  the  player  can  arrange  to  help  seal  a  rift. 

•  The  idea  of  fracture  points  would  also  make  more  sense  if  it  was  this  grid  itself  that 
contained  the  rift  and  therefore  needed  to  be  fixed. 

•  Gaining  credits  for  organizing  planets  seems  like  an  odd  concept.  This  would  make  more  sense 
if  you  simply  had  to  organize  the  nodes  before  attempting  a  seal.  Since  organizing  the  planets 
is  satisfying  and  fun,  this  would  be  a  more  interesting  pushback  than  the  credit  earning. 

•  Could  have  the  nodes  unlock  after  a  failed  seal  attempt  or  when  a  new  rift  appears,  so  that 
there  is  also  a  pushback  to  encourage  the  best  possible  seal  attempts. 

•  Alternately,  this  pushback  could  come  from  giving  the  player  only  a  fixed  number  of  rift 
sealers  to  deploy,  so  that  they  can’t  have  10  or  20  different  star  systems  cooking  at  once. 

•  Another  opportunity  for  progression  here,  where  the  player  earns  new  rift  sealers  in 
some  way.  This  could  also  be  a  good  mechanism  to  introduce  the  concept  of  the 
autopilot  and  engaging  multiple  sealers. 

F.2.3  User  Control  and  Freedom 

Players  often  choose  actions  by  mistake  and  will  need  a  clearly  marked  “emergency  exit’’  to 
leave  the  unwanted  state  without  having  to  go  through  an  extended  dialogue.  Support  undo  and 
redo.  What  mistakes  require  too  much  work  to  recover  from? 
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•  It  is  tedious  and  repetitive  to  have  to  recharge  the  red  bar  after  a  failed  rift  closure  attempt. 
Consider  allowing  multiple  attempts  on  the  same  rift,  or  incorporating  one  of  the  pushbacks 
discussed  above,  to  make  this  time  into  gameplay  instead  of  what  seems  like  tedium. 

F.2.4  Consistency  and  Standards 

Players  should  not  have  to  wonder  whether  different  words,  situations,  or  actions  mean  the 
same  thing.  Follow  platform  conventions.  What  inconsistencies  make  playing  or  understanding 
the  game  harder? 

•  It  was  unclear  why  a  planet  scan  would  reveal  one  enemy,  but  a  second  scan  on  the  same  planet 
would  sometimes  reveal  a  different  enemy  situation  (e.g.,  more,  less,  different  locations). 

•  Consider  showing  aliens  arriving  from  off-screen  every  time  they  are  added. 

•  Red  is  typically  associated  with  bad  things  or  with  health  bars,  so  it  is  strange  to  have  the 
credits  bar  be  red.  It  is  also  inconsistent  with  the  graphics  for  adding  credits  (the  +50  or 
whatever  from  organizing  planets),  which  are  a  more  credit-like  yellow  color. 

F.2.5  Error  Prevention 

Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a  problem  from 
occurring  in  the  first  place.  Either  eliminate  error-prone  conditions  or  check  for  them  and 
present  players  with  a  confirmation  option  before  they  commit  to  the  action  .  Does  the  game  let 
me  blunder  into  serious  errors? 

•  The  meters  concept  discussed  in  the  Visibility  of  System  Status  section  would  help  to  keep  the 
player  from  kicking  off  too  long  of  a  sealing  attempt,  and  to  better  juggle  the  time  an  attempt 
will  take  with  its  potential  for  success. 

F.2.6  Recognition  rather  than  Recall 

Minimize  the  player ’s  memory  load  by  making  objects,  actions,  and  options  visible.  The 
player  should  not  have  to  remember  information  from  one  part  of  the  game  to  another. 
Instructions  for  use  of  the  system  should  be  visible  or  easily  retrievable  whenever  appropriate. 
Which  instructions  are  difficult  to  find  or  refer  to? 

•  No  issues. 

F.2.7  Flexibility  and  Efficiency  of  Use 

Accelerators — unseen  by  the  novice  player-can  speed  up  the  interaction  for  the  expert  player 
so  that  the  game  can  cater  to  both  inexperienced  and  experienced  players.  Allow  players  to 
tailor  frequent  actions.  What  actions  do  I find  repetitive  or  wish  I  could  hotkey/macro? 

•  Navigating  between  the  left  menu,  the  map,  and  the  IDEA  interfaces  requires  considerable 
mouse  movement.  Please  consider  integrating  keyboard  shortcuts  and  moving  some  of  the 
information/interactions  directly  to  the  map  or  IDEA  graph.  For  example,  it  would  be  more 
efficient  to  have  the  Rift  Sealing  button  located  in  the  IDEA  space. 

F.2.8  Aesthetic  and  Minimalist  Design 

Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Every  extra 
unit  of  information  in  a  dialogue  competes  with  the  relevant  units  of  information  and  diminishes 
their  relative  visibility.  What  can  be  removed  to  improve  clarity? 

•  The  interface  seems  like  it  could  be  streamlined  by  combining  the  various  controls  into  a  single, 
size-constrained  location  (as  opposed  to  having  them  spread  across  the  IDEA,  left  menu,  and 
map).  Seek  opportunities  to  create  contextual  controls  from  the  map  area  when  interacting  with 
the  planets  or  aliens. 
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F.2.9  Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors 

Error  or  failure  messages  should  be  expressed  in  plain  language  (no  codes),  precisely 
indicate  the  problem,  and  constructively  suggest  a  solution.  When  I  fail  or  make  a  mistake,  do  I 
understand  why  and  how  to  do  better? 

•  Although  some  text  accompanies  a  failed  rift-sealing  attempt,  it  is  difficult  to  figure  out 
specifically  what  is  being  done  incorrectly.  Better  indication  of  the  relative  strength  of  attempts 
would  help  the  player  learn  the  best  strategies. 

•  The  feedback  for  killing  vs.  hurting  aliens  is  a  bit  confusing  and  unclear,  and  the  use  of  both 
good  and  bad  explosion  graphics  makes  it  even  harder  to  parse.  Consider  using  explosions  for 
only  good  or  only  bad  outcomes  and  presenting  some  other  visual  feedback  for  the  other  type 
of  outcome. 

F.2.10  Help  and  Documentation 

Even  though  it  is  better  if  the  game  can  be  used  without  documentation,  it  may  be  necessary 
to  provide  help  and  documentation.  This  information  should  be  easy  to  search,  list  concrete  steps 
to  be  carried  out,  and  not  be  too  large.  What  things  do  I  wish  I  could  look  up  quickly  while 
playing? 

•  The  help  button  should  provide  more  information  than  to  label  the  lock/unlock  buttons. 

F.3  Game  Questions 
F.3.1  Tutorial  Analysis 

Assessment  of  the  effectiveness  of  the  tutorial;  how  effective  it  is  at  teaching  its  topics,  how 
well  you  feel  you  understand  the  game  at  the  end  of  it,  how  well  it  sets  you  up  for  the  jump  to 
real  gameplay,  how  well  it  draws  you  into  the  game.  Did  the  tutorial  help?  What  kind  of  player 
does  it  support?  Would  you  finish  it? 

•  The  tutorials  could  be  more  specific.  For  example,  it  was  very  helpful  when  a  specific  planet 
was  identified  (e.g.,  select  planet  number  5).  This  type  of  specification  should  be  used 
throughout. 

•  It  was  unclear  how  the  blue  lines  in  the  IDEA  connected  to  the  map  space.  More  emphasis  on 
the  ability  to  select  one  and  see  its  nodes  would  help  clarify  this,  so  perhaps  that  should  be  a 
distinct,  required  step  in  a  tutorial. 

F.3.2  Engagement  Analysis 

Assessment  of  overall  enjoyment,  including:  what  elements  make  the  game  enjoyable,  what 
elements  detract.  How  can  elements  be  changed  to  enhance  enjoyment? 

•  Although  the  game  mechanics  are  tied  to  specific  domain  elements,  there  is  an  opportunity  to 
add  variety  without  affecting  the  underlying  mechanics.  For  example,  by  incorporating  a 
variety  of  enemies  and  weapon  types,  the  player  could  experience  additional  levels  of  strategy 
and  tactics,  but  still  accomplish  the  same  task  behind  the  scenes. 

•  Progression  mechanics  would  help  tie  the  individual  moments  of  play  to  a  larger  goal  and 
feeling  of  accomplishment.  This  could  include: 

•  Unlocking  or  purchasing  weapon,  sensor,  or  defense  upgrades  to  improve  combat. 

•  Improving  the  autopilot/purchasing  additional  sealers  to  allow  more  rifts  to  be  attempted 
at  once  or  to  increase  the  size  of  rift  that  can  be  tackled. 

•  Improving  the  payoff  of  successful  seals  or  alien  kills  through  some  sort  of  multiplier. 
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•  Organizing  nodes  is  fun,  so  some  additional  emphasis  on  it,  either  through  requiring  it  in  order 
to  attempt  a  seal,  or  through  some  payoff  mechanism  from  other  players  (where  you  market 
your  organized  level  or  something)  would  increase  engagement. 

F.3.3  Coherency  Analysis 

Assessment  of  coherency  and  balance  between  elements,  such  as  mechanics,  story, 
aesthetics,  and  technology.  What  elements  seem  to  conflict  or  undermine  each  other? 

•  The  mechanics,  story,  and  aesthetics  seem  to  reinforce  each  other  in  most  ways.  There  are  a 
few  specific  issues  discussed  in  the  Consistency  and  Standards  section. 

F.3.4  Analysis  of  Value  to  Player  Provided  by  Game 

What  is  the  player  getting  out  of playing  the  game?  Why  would  they  press  the  “next  level’’ 
button?  Is  it  fun?  Are  they  making  progress?  Are  they  leveling  up  or  filling  a  bar  or 
something?  Are  their  game-playing  peers  depending  on  them? 

•  Several  of  the  game  elements  are  entertaining  to  play  with,  such  as  organizing  the  nodes, 
finding  a  good  seal  attempt,  and  fighting  off  the  aliens.  However,  the  following  elements  feel 
like  dead-ends: 

•  Fighting  off  the  aliens  is  fun  for  a  bit,  but  without  some  pushback  to  letting  them  attack  or 
some  clear  benefit  from  destroying  them,  it  quickly  loses  interest. 

•  Organizing  the  nodes  is  fun,  but  once  you  realize  that  all  it  gets  you  is  a  bit  easier  of  a  time 
in  combat  (since  you  can  get  the  credits  faster  by  just  exploiting  it),  it  loses  value. 

•  Finding  a  good  spot  to  seal  is  an  interesting  problem,  but  frustrating  when  you  fail  and 
aren’t  sure  if  it  was  because  you  missed  something  or  just  because  this  particular  level  has 
some  quirk. 

•  Finding  ways  to  close  the  loop  on  these  mechanics  and  let  them  feed  into  each  other  or 
themselves  to  improve  future  play  would  dramatically  increase  the  value  to  the  player. 

•  Some  indication  of  global  progress  and/or  leaderboard  rank  would  help  motivate  players  to 
improve  their  own  score. 

•  This  also  might  be  true  for  solutions  to  individual  levels,  if  there’s  some  way  to  optimize 
them,  and/or  for  combat-related  feats. 

•  Achievements  or  some  form  of  high  score,  both  globally  and  personally,  would  help 
players  compare  their  success  to  what  is  possible  and  gain  some  value  from  interacting 
with  what  others  are  achieving. 
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Appendix  G.  CSFV  Marketing  and  Promotional  Materials 


Usability  Testing 


Goal:  First  Make  Games  FUN! 

•  Work  to  create  games  anyone  can  play  with  familiar 
mechanics  and  environments.  Lower  the  learning 
curve  with  common  themes,  simple  commands 
and  easy  interfaces. 

•  Long  tutorials  don't  always  work  to  teach  players 
and  can  lead  to  acquisition  and  retention  problems. 

Show  gameplay  in  short  bursts,  then  allow  players 
to  play  "real"  levels.  Teach  new  mechanics  as  needed. 

Share  Your  Insights  and  Learn  from  Other  Teams 

•  The  Next  Steps  reports  we've  seen  so  far  have  identified  a  number  of  important 
usability  issues  and  have  shared  several  valuable  solutions.  Consider  those 
ideas  and  how  they  can  be  applied  to  your  game. 

r - \ 

201 5  Strategic  Marketing  Plan  Updates 

A  Major  issue  for  players  is  they  don't  see  the  consequences  of  their  play — even  negative  s 
consequences  are  motivators. 

Think  about  core  pillars  of  game  design: 

1 .  Consequences  of  gameplay  actions  are  clear  rewards  or  negative  consequences. 

2.  Belonging/teamwork 

3.  Novelty/surprise 

4.  Mastery — being  able  to  improve  and  master 

Consider  these  pillars  and  similar  concepts  in  next  steps  (usability  testing  review)  docu¬ 
ments. 

Strategy: 

How  is  Design  Club  going?  How  much  fun  are  the  games?  How  easy  are  they  to  get  into  for 
a  new  player?  How  about  obstacles  for  new  players — long  tutorials,  long  learning  curves, 
frustration  factors,  etc.? 


Acquisition  and  Retention — Can  players  get  into  "real"  gameplay  quickly,  or  are  they  stuck 
in  long  tutorials?  Are  all  game  mechanics  being  taught  at  the  start  of  games,  or  when  they 
are  needed? 


Pillars  of  game  design — Consequences  of  actions,  negative  or  positive,  clear  rewards  or 
negative  consequences.  Make  sure  players  know  if  what  they  did  was  bad  or  good. 


Sense  of  belonging — Is  there  a  way  to  integrate  teamwork/social?  Working  toward  a  collec¬ 
tive  goal  is  implied— CSFV.  Can  we  make  this  a  key  component  of  each  gameplay  session? 

Novelty/Surprise — What's  new  and  interesting  as  players  play? 

Mastery — Can  players  improve  as  they  play?  How  do  they  know  they  are  mastering  the 
game? 

_ 
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Messaging 


Key  Message: 

Data  Security  is  Getting  More  Important  All  the  Time 

Key  Call  to  Action: 

Play  Games  to  Help  Make  Data  More  Secure 

•  Due  to  our  communication  constraints,  we  can't  create  traditional  mission  statements  or 
brand  differentiators.  We  need  to  communicate  our  key  messaging  and  call  to  action  in 
our  overall  PR  and  social  media  marketing.  We  need  to  explain  how  players  contribute  to 
science  and  software  security,  i.e.,  what  this  project  does. 

Some  Message  Elements 

•  Infographics — Showing  how  a  game  performs  CSFV. 

•  Text-based — Blog  posts  and  other  articles  describing  how  the  project  works  and  how  the 
development  process  has  evolved. 

•  Hints  &  Tips — Short  blurbs  about  each  game  that  tell  how  they  work  and  contribute  to  the 
project.  Useful  for  in-game  load  screens,  social  media  posts,  community  postings,  etc. 

“Did  you  know  Monster  Proof  solves  23  equations  every  time  you  level  up?" 

“Dynamo  is  calculating  proofs  as  you  make  choices  from  the  Analyzer  Tools." 

"Paradox  is  verifying  software  as  you  select  multiple  nodes  and  run  the  solver." 
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Strategy: 

Infographics — Show  how  games  perform  CSFV.  Are  games  in  a  place  to  create  infograph¬ 
ics?  Are  assets  available?  Is  there  bandwidth  to  provide  assets  and  mockups  with  text?  In¬ 
fographics  can  be  introduced  in  newsletters  ana  other  social  and  press  to  provide  info  and 
create  buzz. 


Blog  Posts — What  are  you  doing  that's  different?  Can  repurpose  scientific  paper  stuff  for 
blog  posts,  blurbs,  short  articles,  captions,  etc.  and  other  material  for  Citizen  Science  page? 

Hints  &  Tips —  Created  from  tutorials  and  rolled  out  at  launches. 
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PR 


Press  Kit — Materials  for  Phase  2  Games 


•  High  Res  Logo 

•  High  Res  Screenshots  (minimum  5) 

•  In-Game  High  Res  Icons,  Characters,  etc. 

•  In-Game  Fonts 

•  In-Game  Sounds  &  Music 


Teaser  Video 

Launch  Press  Release — Written  by  In-House  PR  if  Possible 
Dev  Team  Updated  Contact  Info  &Titles 
Updated  Press  Contact 
Press  Kit  Will  Be  Posted  Online 


Teaser  Ad  Campaign  Before  Launch 

•  Series  of  ads  for  Google  ad  network  and  social  media  to  create 
pre-launch  buzz  about  Phase  2.  Ads  show  the  work  already 
done  by  the  CSFV  program  using  a  comparison  method. 

(A  graphical  representation  of  the  amount  of  data  Verigames 
gameplay  has  analyzed.)  We  first  need  DARPA  permission  to 
release  this  data. 

•  The  tagline  at  the  bottom  of  each  ad  will  be  something  like, 
"Verigames  Phase  2  Games  Are  Poised  to  Achieve  Even  More! 
Get  Ready  to  Play!" 


PLAY  TO  CURE 


GENES  IN  SPACE 


IN  ONE  MONTH 

UIIAtK 

HAVE  ANALYSED 


MILES  OF 
DNA  DATA 


AROUND 


,170 


HO 

TH  S  B  Tt€  L£MG~H  OF  I 

540 


FOOTBAIt  PITCHES 


A  /  VuentHM-uriH 

EMPIRE  STATE  BUILDING 


DATA  ANALYSED  WOULD 


HAVE  TAKEN  A 

SCIENTIST  APPROXIMATELY 


6  MONTHS 


;r  “i  .»t  amd.im  nv  rv 


s 
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Strategy: 

Teaser  Material — Concept  art,  placeholder  art  as  well  as  game  design  comps  and  mockups 
are  all  good  as  teaser  material  and  social  media  marketing.  Need  tone  submitted  to  PAO  so 
can  be  used. 


Ad  Campaign  Above  is  a  Suggestion — Needs  PAO  approval.  Is  it  a  good  idea  in  terms  of 
comparison — that  is,  what  kinds  of  comparisons  can  be  made  of  CSFV  progress  with 
real-world  items?  Verigames  has  analyzed  X  amount  of  data,  etc?  What  kind  of  data  relates 
to  what  game? 


Generate  Excitement  on  Social  Media  and  Press  about  Phase  2 — Need  all  dates  to  create 
press  calendar.  Need  to  create  press  calendar  that  takes  PAO  approval  times  into  account. 


Utilize  All  TA1  Press  Departments  As  Possible — Get  press  releases  written  and  disseminat¬ 
ed  by  SRI,  UCSC,  BBN/Raytheon,  Center  for  Game  Science,  etc.  as  much  as  possible.  Work 
with  these  groups  to  do  all  possible  follow  up  with  existing  press  contacts  for  coverage.  Last 
time  they  made  promises  tney  did  not  keep.  How  can  we  get  them  to  follow  through? 


\ _ 
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PR  (Continued) 


Press  Release  Push  at  Launch 

•  Start  with  traditional  PR — Press  releases  sent  out  to  as  many  relevant  outlets  as  possible 
with  follow-ups  to  encourage  coverage. 

•  Support  with  ongoing  social  media  marketing — Posts  of  in-game  content,  blogs,  media 
coverage  and  relevant  related  content  across  social  media. 

Advertising — Google  Networks  and  Social  Media  Boosts 

•  Currently  gathering  advertising  intelligence  using  small  ad  buys  (October  through  April). 
Small  buys  of  Adwords,  static  and  animated  banners,  seeing  what  works  best  to  improve 
our  key  conversion  metric — new  players. 

•  Also  watching  social  media  traffic,  site  traffic,  community  engagement  and  new 
registrations  as  additional  information  on  what  advertising  is  working  in  what  markets. 

•  Will  do  bigger  Phase  2  ad  buys  at  launch  using  gathered  advertising  intelligence. 


f - \ 

2015  Strategic  Marketing  Plan  Updates 

Strategy: 

Asking  all  companies  with  in-house  PR  to  write  and  send  their  own  releases  to  their  own 
press  contacts  for  maximum  coverage  for  investment.  We  will  support  others  as  needed,  en¬ 
suring  no  PR  overlap. 

Need  high  quality  art,  logos,  taglines  and  other  game  info  to  create  ads  that  work. 


Work  on  Ads  for  Google  Network — Get  that  going  with  TopCoder/Appirio.  Games  that  have 
relevant  assets,  start  creating  banners  (animated  up  to  1 50K  at  sizes  we  tested). 


Check  In  on  Ad  Budget,  Method  of  Payment,  Etc.  Based  on  2014Tests — Never  received 
relevant  social  media  traffic,  site  traffic,  community  engagement  analytics.  Ask  for  that  ma¬ 
terial  from  Appirio. 


v _ 
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CITIZEN  SCIENTIST  RESOURCES 

ffl  f  *  G 

Citizen  Science 

HOME  MOG  NTERYfWS 

MEDIA  ARTICLES 

WEMAPHICS 

New  verigames.com  Citizen  Science  Webpage 


Will  contain  modules 
with  several  forms  of  CSFV 
scientific  information  to 
accommodate  different  ways 
people  take  in  information 
and/or  personal  preference. 

We  want  users  to  be  able 
to  find  the  info  they  want  in 
a  form  they  can  absorb.  If 
users  want  to  go  deeper — 
the  resources  are  available. 

•  Blog  Posts 

•  Images  with  Captions 

•  Blurbs 

•  Media  Articles 

•  Infographics 

•  Scholarly  Articles 

•  Bibliography 

•  Video  &  Podcasts 


2015  Strategic  Marketing  Plan  Updates 


Strategy: 

Citizen  Science  Web  Page — What's  the  status?!  Need  to  know  ASAP.  Citizen  Science  is  vital 
to  phase  2  marketing  strategy!  Working  with  TopCoder  web  design  to  best  integrate  head¬ 
er  into  overall  site  so  it's  easily  accessible  throughout  without  a"pasted-on"look  and  feel. 
Working  to  reduce  bounce  rate  and  increase  key  conversion  metric — game  players — by  pro¬ 
viding  info  people  are  looking  for — without  the  clicking  around  heatmaps  we  saw  before. 
May  use  A/B  testing  to  optimize  best  methodology.  Can  do  usability  testing  on  website  to 
obtain  user  input. 


Verigames.com — What's  up  with  the  phase  2  website?  Need  to  get  an  update  from  Appirio 
ASAP.  Excellent  usability  on  verigames.com  and  optimized  implementation  of  Citizen  Sci¬ 
ence  section  are  vital  to  phase  2  marketing  strategy.  Very  concerned  that  design/imple¬ 
mentation  holdups  will  cause  problems  executing  marketing  strategy  and/or  will  cause  the 
same  major  problems  of  phase  1 — will  make  Verigames  not  Took  like  a  going  concern.  In 
phase  1,  the  website  was  "Coming  Soon"for  a  year!  My  direction  took  months  to  be  imple¬ 
mented  and  75%  of  was  never  implemented  with  no  explanation.  If  that  happens  in  phase 
2,  users  will  again  come  to  a  website  that  looks  like  it  is  not  happening,  which  will  again 
lead  to  a  significant  volume  of  bounces  before  players  even  try  games.  Plus,  press  outlets 
will  not  give  us  a  second  chance  for  coverage  if  we  can't  get  the  site  going  1 8  months  later. 
Concerned  about  Wiki — most  recent  content  nearly  500  days  old. . .  ,7About  Us, ""FAQ"  is  from 
201 3  and  is  still  referencing  TopCoder  as  legal  entity  (prime  contractor).  Forums  and  game 
comments  are  all  many  months  old. . . 


_ _ 
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Community 


•  Introduce  community  integration  into  games  as  rewards — Naming  rights  of  levels 
and  other  in-game  artifacts. 

•  Add  friendly  competitions  and  other  social  in-game  interactions. 

•  Integrate  pop-ups  that  come  up  after  a  few  moves,  every  level,  every  few  minutes,  etc. 
that  encourage  guest  players  to  register  so  they  can  become  part  of  the  community. 


•  The  verigames.com  Forums  section  has  been  revamped  so  it's  better  organized, 
easier  to  use  and  more  attractive,  making  it  more  useful  for  community  members 
and  more  welcoming  to  new  players. 


NWttwnff/forrrr 

a  m 


Encourage  new  people 
to  become  part  of  the 
community  through 
direct  interaction  between 
players  and  developers — 

Encourage  players  to  post 
questions,  host  live  chats 

with  devs,  play  live  game  demos,  tournaments,  etc. 


iJLl 


KM  » 
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Strategy: 

There  has  been  a  lot  of  talk  about  integrating  community  gameplay  mechanics  options 
among  game  Design  Club,  other  game  design  conversations  and  past  PI  meetings.  It's  time 
to  begin  real  integration  of  community  mecnanics  across  and  between  games.  I  would  like 
to  see  community  features  tested  in  our  next  usability  testing  session  and  potentially  A/B 
testing  on  the  verigames.com  website  using  analytics. 


Developer  Interaction  with  Player  Community — Plan  on  direct  developer  interaction  with 
player  community  at  least  once  a  month  starting  at  launch.  Can  be  AMAs,  tournaments, 
chats,  etc. 


Overall  Community  in  Games — What  social  integration  is  available  in  your  game?  Rewards? 
Competition?  Registration  encouragement?  Interaction  between  players  and  developers? 
What's  your  preference?  What's  best  for  the  title?  Can  you  partner  with  another  game  (or  all 
the  games?)  to  create  group  interaction  among  players  to  get  players  to  try  other  games? 


V _ 7 
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Retention  Strategies 


•  Player  Bonuses — In-game  score  or 
other  bonus  on  a  timer  that  lasts 
longer  than  the  average  play  session. 

Players  come  back  for  the  bonus. 

•  Daily  Bonuses — Players  comeback 
to  the  game  to  get  the  bonus. 

•  Implement  in-game  progress  bars 
that  indicate  player  progress  in  CSFV. 

•  Create  infographics  about  each 
game's  gameplay  mechanics  (directed 
by  development  teams,  created  byTopCoder/Appirio). 

•  Create  a  video  walk-through/tutorial  for  each  game  before  launch. 
Tutorials  should  be  short  and  simple.  It's  better  to  have  more  than  one 
video — perhaps  illustrating  a  particular  gameplay  mechanic  or  section 
of  the  Ul — instead  of  a  single  long  tutorial  video.  For  wide-ranging 
tutorial  examples,  check  Twitch  and  YouTube. 
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Key  motivators  that  act  as  passive  "consequences"  of  player  non-actions  have  been  proven 
to  work  across  casual,  social,  mobile  and  free-to-play. 

Offer  multiple  game  learning  strategies.  Folks  refuse  to  read  and  are  saying  tutorials  are 
too  long  in  usability  tests.  Yet,  they  need  to  learn  the  games  somehow.  Allow  them  to  pick 
up  and  play — and  go  back  and  learn  more  later  once  they're  interested  and  invested  in  the 
game  with  bonus  points. 

Strategy: 

Retention — Daily  bonuses? Times  bonuses?  CSFV  progress  bars?  Video  walkthroughs? 


Multiple  Learning  Strategies — Can  pick  up  and  play,  earn  bonuses  and  then  learn  key  me¬ 
chanics  once  invested. 


%  1  r 

BONUS 

POINTS 
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Retention  Strategies  (continued) 


•  Notifications — Non-annoying 
messages  sent  through  social  media. 

Provide  science  info  about  CSFV  or 
other  relevant  science  info  of  the  day. 

"Verigames  congratulates  the  Rosetta 
Mission  on  its  historic  achievement!" 

(with  embedded  linkfor 
users'  convenience). 

•  A  lot  of  players  are  in  fields  similar  to  CSFV.  Target  those  fields  by  providing 
information  of  interest  to  them. 

•  People  in  the  field  sometimes  don't  want  "fluffy"  games.  (The  Xylem  team  learned 
some  of  their  players  just  wanted  the  math  and  wanted  to  get  rid  of  the  island  story 
of  the  game.)  Create  settings  that'll  allow  players  to  turn  off  the  "fluffy  stuff." 

•  The  default  should  be  games  that  are  accessible  to  the  widest  possible  audience. 
Provide  pop-ups  informing  players  of  available  settings. 


10 
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Settings  can  also  be  a  pre-cursor  to  Mechanical  Turk  games  (turn  off  everything  for  MT  ver¬ 
sion?).  Can  make  game  simpler  without  "fluff"  for  serious  types. 

Notifications  that  provide  information  that's  useful/interesting  can  remind  players  of  the 
games  while  actually  providing  content  of  use.  Must  be  careful  not  to  endorse  according  to 
DARPA  guidelines — scientific,  educational,  historic,  okay.  Product,  brand,  celebrity,  etc.,  not 
okay. 

Strategy: 

Retention — Social  media  messages  about  science  in  general  and  CSFV,  targeting  similar 
science  fields. 


v _ / 


■  Philae  Lander  >±  Follow 

OPtritoaQOM 

Touchdown!  My  new  address:  67P! 
ifCometLanding 

7M  2.868 

ItMini.  19Nov9014 
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Appendix  H.  Verigames  Contest  Rules 

Within  the  DARPA  Crowd  Sourced  Formal  Verification  (CSFV)  Program,  the  Defense 
Advanced  Research  Projects  Agency  (DARPA),  Air  Force  Research  Laboratory  (AFRL), 

Charles  River  Analytics  Inc.,  and  TopCoder,  Inc.  will  be  holding  a  contest  to  reward  high- 
scoring  players  across  each  of  the  five  games  with  community  events  and  prizes.  During  the  next 
four  weeks,  we  will  track  game  scores  across  all  players  in  the  community,  and  across  all  games. 
For  each  game,  we  will  provide  rewards  to  the  top  5  scorers  as  outlined  below: 

•  First  Prize:  The  top  scorer  from  each  game  will  be  invited  to  participate  in  a  live  web 
conference  with  game  developers  and  DARPA  representatives  from  the  CSFV  program.  This 
conference  will  start  with  a  30-minute  presentation  from  DARPA  describing  the  goals  and  the 
approach  of  the  program,  followed  by  a  30-minute  presentation  from  the  game  developers 
describing  the  approach  of  their  games.  Finally,  it  will  end  with  a  30-minute  chat  session  in 
which  developers  will  answer  questions  from  the  winners.  Winners  of  the  first  prize  will  gain 
a  special  tag  to  their  account  identifying  them  as  game  gurus,  linked  to  any  of  their  posts  on 
the  forum  and  their  participation  in  live  chat  features  for  each  of  the  games. 

•  Second  Prize:  The  second  and  third  scorer  from  each  game  will  have  access  to  a  recorded 
segment  of  the  web  conference  described  above,  focusing  on  the  first  hour  of  the  session  (the 
30  minute  DARPA  presentation  and  the  30-minute  game  developer  presentation).  Winners  of 
the  second  prize  will  also  have  access  to  the  third  prize  as  well.  Winners  of  the  second  prize 
will  gain  a  special  tag  to  their  account  identifying  them  as  game  experts,  linked  to  any  of  their 
posts  on  the  forum  and  their  participation  in  live  chat  features  for  each  of  the  games. 

•  Third  Prize:  The  fourth  and  fifth  scorer  from  each  game  will  gain  an  honorable  mention  tag 
to  their  account  identifying  them  as  strong  contributors,  linked  to  any  of  their  posts  on  the 
forum  and  their  participation  in  live  chat  features  for  each  of  the  games. 


Contest  rules  and  details: 

•  To  participate,  you  must  have  a  tracked  account  on  Verigames.com;  anonymous  players  will 
not  be  eligible  for  any  rewards. 

•  Winners  will  be  responsible  for  their  own  network  fees  to  participate  in  the  web  conference 
and/or  to  access  videos  of  that  conference;  DARPA,  Verigames,  TopCoder,  and  Charles  River 
Analytics  bear  no  responsibility  to  provide  or  maintain  connections  to  support  participation. 

•  The  contest  will  be  conducted  July  13,  2015  through  August  14,  2015.  Only  scoring  accrued 
during  that  period  will  be  considered  as  part  of  the  performance  assessment  entering  into  the 
contest. 

•  The  date  of  the  community  event  is  to  be  detennined,  but  will  occur  within  4  weeks  after  the 
end  of  the  contest. 

•  Questions  for  the  development  team  and  DARPA  must  be  submitted  and  approved  at  least  one 
week  prior  to  the  targeted  chat. 

•  All  prizes  are  considered  to  have  no  monetary  value.  Prizes  are  intended  for  the  community 
members  who  win,  and  will  not  be  transferable  to  other  community  members  or  non¬ 
community  members. 

•  The  prizes  listed  above  are  considered  to  have  no  monetary  value;  winners  will  receive  no 
monetary  rewards  based  on  this  contest. 
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You  must  be  18  years  or  older  to  win.  Void  where  prohibited. 


Distribution  Statement  A:  Approved  for  public  release;  distribution  unlimited 
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Appendix  I.  Usability  Study:  Dynamakr 

1.1  Objectives 

These  games  are  made  to  be  somewhat  complex,  as  the  movements  and  objectives  help  to 
solve  complex  mathematical  problems.  The  key  business  objectives  are  to  review: 

1.  Tutorial  effectiveness  in  onboarding 

2.  Gameplay  mechanics  and  impacts  on  retention 

3.  Guidance  and  direction  support 

4.  Engaging  and  lasting  gameplay 

1.2  Demographics 

The  population  consisted  of  10  users  who  were  reasonably  distributed  across  the  United 
States.  Most  subjects  were  25-44  years  old,  with  one  in  the  18-24  range  and  one  in  the  45-64 
range.  The  population  was  relatively  evenly  split  in  gender  (6  female,  4  male).  All  subjects  had  a 
household  income  between  $30,000  and  $99,000. 

1.3  Behavior  Metrics 


They  can  do  what  they're 
told,  but  don't  know  why 

100% 


...were  able  to  accomplish/ 
succeed  through  the  initial 
tutorial,  but  didn't  really 
know  the  reasons  for  the... 


End-Goal  objectives  and 
purpose  NEEDED 

50% 


What  does  a  good  pattern 
look  like?  What  exactly  are 
they  trying  to  accomplish  in 
the  Dynamo?  Why  is  building 
energy ... 


Mechanics  come  off  as 
passive  clicking  and  waiting 


70% 


Felt  the  mechanics  were  too 
passive  or  based  on  chance, 
rather  than  any  strategy. 


Demonstration,  rather  than 
instruction,  are  most  helpful 


30% 


Hover  tips  routinely 
mentioned.  Tutorial  should 
really  include  next  steps  for 
complicated  parts. 


* 


Test:  Going  through  the 
Dynamo 

10% 


Larger  game  purpose  and 
design  raises  curiosity 


30% 


Praised,  and  implied  the 
initial  allure  of  the  game 
could  lead  to  playing  more. 


1.4  Key  Findings 

Key  findings  included  the  following: 

1 .  Initial  intro  and  design  allures  people  to  potential  of  engaging  gameplay.  Maintain  sound  effects 
and  galactic  scale. 

2.  High  success  with  building  patterns,  some  miss  energy  building  requirement. 

3.  What  makes  a  pattern  better  than  others?  Why  would  you  load  up  on  the  right,  but  not  on  the  left? 

4.  Unless  paying  close  attention,  it  is  very  easy  to  miss  how  pattern  building  relates  to  what  the 
Dynamo  experience  will  be  like.  This  is  pivotal.  Should  players  be  utilizing  strategy  when  building 
patterns  as  a  way  to  impact  the  Dynamo  shoot-out?  If  so,  communicate  earlier. 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 

77 


5.  Make  certain  that  Go  Dynamo!  and  Continue  buttons  are  large  and  noticeable,  some  missed  them 
initially  which  caused  delays.  Potentially,  just  remove  any  need  to  click  to  the  next  space,  as  long 
as  there  is  some  transition  demo  to  introduce  the  Dynamo  and  levels. 

6.  Majority  suggest  leveraging  more  tooltips  and  hover  instructions  when  doing  certain  interactions, 
rather  than  large  chunk  of  text  before  and  after  parts  of  the  tutorial. 
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Appendix  J.  Usability  Study:  Binary  Fission 
J.1  Objectives 

These  games  are  made  to  be  somewhat  complex,  as  the  movements  and  objectives  help  to 
solve  complex  mathematical  problems.  The  key  business  objectives  are  to  review: 

1.  Tutorial  effectiveness  in  onboarding 

2.  Gameplay  mechanics  and  impacts  on  retention 

3.  Guidance  and  direction  support 

4.  Engaging  and  lasting  gameplay 

J.2  Demographics 

The  population  consisted  of  1 1  users  who  were  reasonably  distributed  across  the  United 
States.  Most  subjects  were  25-44  years  old,  with  some  in  the  18-24  range,  and  one  in  the  45+ 
range.  The  population  was  relatively  evenly  split  between  males  and  females.  Most  subjects  had 
a  household  income  between  $40,000  and  $99,000,  with  several  in  lower  income  brackets. 

J.3  Behavior  Metrics 


Early  engagement:  Early  confusion 

Smooth  entry 

36.36%  63.64% 


Pleasing  graphics  and  clear  Participants  are  confused 

instructions  about  how  to  play  the  game 


A  little  bit  more  clarity 
would  help. 

27.27% 


...desired  more  clarity  into 
the  larger  objective,  a  demo 
video,  or  better  filter 
understanding. 


n 


^suggestion  I  Prompt  5:  4.  5.  6 


Lobby  screen:  A  bit  chaotic, 
color  coding  helps  slightly 


27.27% 

Could  utilize  a  bit  more 
direction  to  motivate  players 
to  go  either  which  way 


□ 


Confidence  builds  slowly,  difficult 
levels  need  some  becking 


100% 


Once  the  levels  get  tough, 
understanding  the  simple 
filtering  mechanics  aren't 
enough  to  truly  succeed. 


J.4  Key  Findings 

Key  findings  included  the  following: 

1 .  Simplicity  of  the  game  leads  to  fairly  successful  ramp-up,  but  a  video  demonstration  or  light  back- 
story  would  increase  engagement 

2.  Minimal  feedback  during  filter  splitting  leads  to  doubt.  Utilize  sound  or  color  feedback  to  increase 
awareness  of  success  or  failure 

3.  Once  the  levels  get  tough,  understanding  the  mechanics  builds  confidence,  but  -50%  of  the 
participants  wanted  something  more  to  work  towards.  Some  assumed  that  the  correct  filter  path 
created  a  larger  pattern  or  form,  but  this  was  not  ever  confirmed. 

4.  Design  mostly  pleases.  Largest  gap  in  elements  is  how  score  is  calculated. 

5.  Increase  the  competitive  nature  through  friend  referral  and  lvl  competition  of  the  same  levels. 

6.  Most  participants  could  have  used  a  slightly  longer  transition  to  harder  levels  while  mechanics  are 
given  feedback  and  the  score  is  explained. 
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Appendix  K.  Usability  Study:  Monsterproof 
K.1  Objectives 

These  games  are  made  to  be  somewhat  complex,  as  the  movements  and  objectives  help  to 
solve  complex  mathematical  problems.  The  key  business  objectives  are  to  review: 

1.  Tutorial  effectiveness  in  onboarding 

2.  Gameplay  mechanics  and  impacts  on  retention 

3.  Guidance  and  direction  support 

4.  Engaging  and  lasting  gameplay 

K.2  Demographics 

The  population  consisted  of  10  users  who  were  reasonably  distributed  across  the  United 
States.  Most  subjects  were  25-44  years  old,  with  one  in  the  18-24  range  and  one  in  the  45-64 
range.  The  population  was  mostly  female  (7  participants).  Most  subjects  had  a  household  income 
between  $30,000  and  $79,000,  with  two  in  lower  income  brackets,  and  one  in  a  higher  bracket, 
and  one  unspecified. 

K.3  Behavior  Metrics 


How  they  approach  the  Tutorial  Navigation 

increase  in  challenge...  Critique 


Confusing  tutorial. 


First  tutorial  leaves  them 
very  confident 


Hinting  towards  difficulty 
during  Tutorial  2 


80% 


40% 


40% 


60% 


1 00% 


...felt  confused  when  the 
game  suddenly  increased 
gears.  Msp  felt  too  rigid,  but 
collection  of  tools  and 
early... 


NEW'  Please  edit  and 
regenerate  the  highlight  reel. 


Participants  wants  to  access 
tutorial  again  once  in-game. 


1  -3  levels  far  too  confusing 
for  difficulty  ahead.  How  are 
letter  grades  assigned? 


Still  easy  for  most.  Hidden 
interactions  still  confuse. 
Many  feel  harder  levels 
coming. 


Mixed  thoughts  about  the 
Elective  vs.  Required  tutorials 


Playing  through  the 
quarry 


What  additional  help  is 
needed?  Tips  while  playing 


How  do  they  feel  about  the 
time  spent  on  the  game? 


The  dissatisfied  share  their 
confusion  and  suggestions 


1 00%  1 00% 


100%  100% 


90% 


It  may  be  better  to  cover 
•essentials*  first,  without 
option  for  bonus  electives 
after? 


Low  retention  for  certain 
difficult  aspects 


Help  center  accessible  within 
the  levels 


For  the  complexities,  they 
feel  more  time  would  be 
needed  to  learn... 


Tutorial  includes  a  lot  -  break 
down  the  larger  proofs  into 
smaller  parts,  which  will 
smooth  out  the... 


K.4  Key  Findings 

Key  findings  included  the  following: 

1.  Difference  between  first  proof  and  later  equations  scales  rapidly  with  little  opportunity  for 
confidence  building. 

2.  Tutorials  add  up  for  too  much  length  -  requires  a  lot  of  memory  retention. 

3.  Very  important  to  give  feedback  when  proper  balance  of  monsters  initiated  -  do  not  wait  for  player 
to  test  equation. 
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4.  Map  felt  too  rigid  for  some,  but  collection  of  tools  and  early  challenge  excited  many. 

5.  Utilize  the  difficulty  of  tutorials  2-3  for  a  bit  longer.  Very  easy  levels  over  and  over  to  build 
confidence,  gain  some  tools,  and  introduce  the  other  game  mechanics  while  players  do  not  lose 
faith. 
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Appendix  L.  Usability  Study:  Paradox 
L.1  Objectives 

These  games  are  made  to  be  somewhat  complex,  as  the  movements  and  objectives  help  to 
solve  complex  mathematical  problems.  The  key  business  objectives  are  to  review: 

1.  Tutorial  effectiveness  in  onboarding 

2.  Gameplay  mechanics  and  impacts  on  retention 

3.  Guidance  and  direction  support 

4.  Engaging  and  lasting  gameplay 

L.2  Demographics 

The  population  consisted  of  10  users  who  were  reasonably  distributed  across  the  United 
States.  Most  subjects  were  25-44  years  old,  with  two  in  the  18-24  range.  The  population  was 
mostly  female  (7  participants).  Most  subjects  had  a  household  income  between  $40,000  and 
$99,000,  with  two  in  lower  income  brackets,  and  one  in  a  higher  bracket. 

L.3  Behavior  Metrics 


Confusion  that  occurred 
in  the  initial  tutorial 

30% 


Early  suggestions  after 
very  initial  gameplay 

40% 


Feedback  about 
mechanics  of  the  game 

70% 


Participants  praise  the 
mechanics 

30% 


Intro  piques  their  interest, 
with  early  confidence. 

40% 


...concept  behind  the  brush 
types  and  why  constraints 
need  to  be  cleared  would 
help  greatly. 


...were  satisfied  with  how  the 
game  was  introduced.  Some 
feel  there  needs  to  be  some 
larger  goal  or  excitement... 


Participants  play  through 
the  harder  levels 

1 00% 


Softer  suggestions  to  whats 
needed  in  the  tutorials. 

30% 


Easier  level  jumping  and  vivid 
design  would  motivate  further 

1 00% 


Highlight/metric  of  those  that 
rated  confidence  at  3-below 

50% 


Optimistic  players  give 
feedback  near  at  the  end 

50% 


...had  much  more  difficulty  in 
the  later  levels  than  they 
anticipated. 


...felt  that  a  tiny  bit  more  ...hinted  at  continued  playing, 

guidance  would  go  a  really  It  is  slightly  bland,  in  a  world 

long  way.  of  over-stimulation,  but 

some  imply... 


Game  fits  well  for  mobile 
platform:  Participants  want 
competitive  side  nurtured  a 
bit  further.  How  to  hit  1 00% 

to... 


L.4  Key  Findings 

Key  findings  include  the  following: 

1.  Tutorial  instructions  too  distant  from  paintbrush  actions  and  paintbrush  type;  difficult  to  recall 
strengths  of  paintbrushes. 

2.  When  new  paintbrushes  are  introduced,  animation  demo  of  how  they  differ  would  help  greatly. 
Along  with  the  ability  to  constantly  be  reminded  via  hover  or  name  of  brush  next  to  brush  image. 
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3.  Design  palette  slightly  soothing,  a  bit  polarizing:  too  dull  for  some.  But  for  several,  it  achieves  a 
nice  tone. 

4.  Scale  and  scope  of  map  size  still  a  bit  jarring  when  later  levels  are  played.  Playing  a  map 
incrementally,  starting  from  further  zoomed  in  on  the  difficult  areas,  may  lead  to  a  better 
engagement  curve. 
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Appendix  M.  Usability  Study:  Ghost  Map  -  Hyperspace 
M.1  Objectives 

These  games  are  made  to  be  somewhat  complex,  as  the  movements  and  objectives  help  to 
solve  complex  mathematical  problems.  The  key  business  objectives  are  to  review: 

1.  Tutorial  effectiveness  in  onboarding 

2.  Gameplay  mechanics  and  impacts  on  retention 

3.  Guidance  and  direction  support 

4.  Engaging  and  lasting  gameplay 

M.2  Demographics 

The  population  consisted  of  10  users  who  were  mostly  distributed  in  the  Eastern  United 
States.  Most  subjects  were  25-44  years  old,  with  two  in  the  45-64  range.  The  population  was 
mostly  male  (7  participants).  Most  subjects  had  a  household  income  between  $40,000  and 
$99,000,  with  two  in  lower  income  brackets. 

M.3  Behavior  Metrics 


Early  thoughts  about 
game:  confusion 


Thoughts  about  playing 
the  game:  confusion 


Understanding  of 
'Combat  Attacking" 


How  engaged  were 
participants? 


Confusion  about  game 
mechanics 


70%  60%  60% 


20%  60% 


confused  participants  Confused  participants  -  confused  participants 

purpose  not  felt.  More  active 
instructions. 


confused  participants;  they  Confusion  about:  rift  sealing, 
want  better  instructions  help  button,  instructions,  and 

the  goal  of  the  game 


Participants  asking  for  better  What  would  raise  the  How  would  they  improve 

instructions  for  game  play  likelihood  of  them  to  play?  the  tutorial? 


40% 


50% 


100% 


They  want  a  video  tutorial 


o 

Create  custom 
metric 


M.4  Key  Findings 

Key  findings  included  the  following: 

1.  Intro  voiceover  should  contain  a  video  of  some  sort  to  excite  and  engage.  Voiceover  easy  to  click 
through  and  miss. 

2.  Greatest  confusion  is  around  Rift  Sealing.  Why  must  they  use  the  bottom  right  sliders,  rather  than 
interacting  in  the  primary  map  area? 

3.  Participants  did  not  seem  to  grasp  the  connection  between  Combat  and  Rift  sealing.  How  does  a 
good  seal  relate  to  killing  the  bugs  and  locking? 
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4.  Locking  a  map  in  and  of  itself  is  not  made  entirely  clear.  Participants  achieve  this,  but  is  it  just  to 
keep  the  map  clean?  Or  serve  a  larger  purpose? 

5.  Feedback  for  bug  health,  along  with  health  vials  on  the  left  side  of  the  game,  not  given  clear  enough 
feedback/values. 
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